Long timeout and finally failing when installing under Docker container

[laeti-tia]

Most of the times, but not everytime, when installing the perfsonar-toolkit on Debian or Ubuntu host running in a Docker container (using our QA scripts), the perfsonar-grafana-toolkit hangs for some longish time and then fail. Display is the following:

Setting up perfsonar-grafana-toolkit (5.1.1-1) ...
Error talking to grafana, waiting
Error talking to grafana, waiting
(and more… and then failing after more than an hour)

Processes running:

root       47869  0.0  0.0   3896  2816 pts/0    S+   14:59   0:00 /bin/bash /var/lib/dpkg/info/perfsonar-grafana-toolkit.postinst configure
grafana    47873  0.5  0.7 1571724 173908 ?      Ssl  14:59   0:05 /usr/share/grafana/bin/grafana server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg
root       47874  0.0  0.0  27224 22476 pts/0    S+   14:59   0:00 python3 /usr/lib/perfsonar/grafana/grafana_folder_permissions.py e65dd0fc-fd6d-45c2-a8bb-f6df1186cf66 d94dcbb1-4076-4c61-b947-3194b38
grafana

It looks like the Grafana API is not responding. However restarting the service while in this loop, doesn't solve the issue.

root@83d65c1db547:/# systemctl status grafana-server
* grafana-server.service - Grafana instance
     Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-07-04 14:09:57 UTC; 1h 13min ago
       Docs: http://docs.grafana.org
   Main PID: 47873 (grafana)
        CPU: 6.103s
     CGroup: /system.slice/grafana-server.service
             |-47873 /usr/share/grafana/bin/grafana server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
             `-47882 /var/lib/grafana/plugins/grafana-opensearch-datasource/gpx_opensearch-datasource_linux_amd64

Jul 04 14:09:59 83d65c1db547 grafana[47873]: logger=sqlstore.transactions t=2024-07-04T14:09:59.006143885Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked"
Jul 04 14:09:59 83d65c1db547 grafana[47873]: logger=folder-service t=2024-07-04T14:09:59.064408058Z level=warn msg="failed to parse user ID" namespaceID=user userID=0 error="identifier is not initialized"
Jul 04 14:09:59 83d65c1db547 grafana[47873]: logger=provisioning.dashboard t=2024-07-04T14:09:59.105392501Z level=info msg="finished to provision dashboards"
Jul 04 14:09:59 83d65c1db547 grafana[47873]: logger=grafana-apiserver t=2024-07-04T14:09:59.469106149Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager"
Jul 04 14:09:59 83d65c1db547 grafana[47873]: logger=grafana-apiserver t=2024-07-04T14:09:59.470309959Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager"
Jul 04 14:10:03 83d65c1db547 grafana[47873]: logger=grafana.update.checker t=2024-07-04T14:10:03.929461596Z level=error msg="Update check failed" error="failed to get stable version from grafana.com: Get \"https://grafana.com/api/grafana/versions/stable\": dial tcp: lookup grafana.com: Try again" duration=5.004158842s
Jul 04 14:10:47 83d65c1db547 grafana[47873]: logger=infra.usagestats t=2024-07-04T14:10:47.93337095Z level=info msg="Usage stats are ready to report"
Jul 04 14:19:58 83d65c1db547 grafana[47873]: logger=cleanup t=2024-07-04T14:19:58.696919288Z level=info msg="Completed cleanup jobs" duration=41.908103ms
Jul 04 15:14:27 83d65c1db547 grafana[47873]: logger=sqlstore.transactions t=2024-07-04T15:14:27.643578332Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked"
Jul 04 15:19:27 83d65c1db547 grafana[47873]: logger=cleanup t=2024-07-04T15:19:27.600477018Z level=info msg="Completed cleanup jobs" duration=14.321696ms
root@83d65c1db547:/# systemctl restart grafana-server
root@83d65c1db547:/# systemctl status grafana-server
* grafana-server.service - Grafana instance
     Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-07-04 15:24:10 UTC; 9s ago
       Docs: http://docs.grafana.org
   Main PID: 48038 (grafana)
        CPU: 1.139s
     CGroup: /system.slice/grafana-server.service
             |-48038 /usr/share/grafana/bin/grafana server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
             `-48047 /var/lib/grafana/plugins/grafana-opensearch-datasource/gpx_opensearch-datasource_linux_amd64

Jul 04 15:24:10 83d65c1db547 grafana[48038]: logger=grafanaStorageLogger t=2024-07-04T15:24:10.82912039Z level=info msg="Storage starting"
Jul 04 15:24:10 83d65c1db547 grafana[48038]: logger=http.server t=2024-07-04T15:24:10.82954238Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl=/grafana socket=
Jul 04 15:24:10 83d65c1db547 grafana[48038]: logger=ngalert.state.manager t=2024-07-04T15:24:10.876457805Z level=info msg="State cache has been initialized" states=0 duration=48.32335ms
Jul 04 15:24:10 83d65c1db547 grafana[48038]: logger=ngalert.scheduler t=2024-07-04T15:24:10.877360781Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1
Jul 04 15:24:10 83d65c1db547 grafana[48038]: logger=ticker t=2024-07-04T15:24:10.877472475Z level=info msg=starting first_tick=2024-07-04T15:24:20Z
Jul 04 15:24:10 83d65c1db547 grafana[48038]: logger=provisioning.dashboard t=2024-07-04T15:24:10.880382687Z level=info msg="starting to provision dashboards"
Jul 04 15:24:10 83d65c1db547 grafana[48038]: logger=provisioning.dashboard t=2024-07-04T15:24:10.903750621Z level=info msg="finished to provision dashboards"
Jul 04 15:24:11 83d65c1db547 grafana[48038]: logger=grafana-apiserver t=2024-07-04T15:24:11.343785477Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager"
Jul 04 15:24:11 83d65c1db547 grafana[48038]: logger=grafana-apiserver t=2024-07-04T15:24:11.344393834Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager"
Jul 04 15:24:15 83d65c1db547 grafana[48038]: logger=grafana.update.checker t=2024-07-04T15:24:15.829031786Z level=error msg="Update check failed" error="failed to get stable version from grafana.com: Get \"https://grafana.com/api/grafana/versions/stable\": dial tcp: lookup grafana.com: Try again" duration=5.00715375s

Still looping on Error talking to grafana, waiting