Persist dashboards created by Grafana users

[kaazoo]

Whenever the Grafana pod is recreated (e.g. during rolling update of k8s cluster), it looses all dashboards that have been created by Grafana users. The new Grafana pod only has the default dashboards.

What's strange is that the alarm definitions seem to be still available:

# kubectl logs  grafana-588cdcbfbc-gd2md -n monitoring grafana
t=2018-07-02T10:24:11+0000 lvl=info msg="Starting Grafana" logger=server version=5.1.1 commit=dc11bf702 compiled=2018-05-07T11:50:10+0000
t=2018-07-02T10:24:11+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2018-07-02T10:24:11+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_USER=grafana_admin"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_PASSWORD=*********"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_AUTH_ANONYMOUS_ENABLED=false"
t=2018-07-02T10:24:11+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_AUTH_BASIC_ENABLED=true"
t=2018-07-02T10:24:11+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2018-07-02T10:24:11+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2018-07-02T10:24:11+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2018-07-02T10:24:11+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2018-07-02T10:24:11+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2018-07-02T10:24:11+0000 lvl=info msg="App mode production" logger=settings
t=2018-07-02T10:24:11+0000 lvl=info msg="Initializing DB" logger=sqlstore dbtype=sqlite3
t=2018-07-02T10:24:11+0000 lvl=info msg="Starting DB migration" logger=migrator
t=2018-07-02T10:24:11+0000 lvl=info msg="Executing migration" logger=migrator id="copy data account to org"
t=2018-07-02T10:24:11+0000 lvl=info msg="Skipping migration condition not fulfilled" logger=migrator id="copy data account to org"
t=2018-07-02T10:24:11+0000 lvl=info msg="Executing migration" logger=migrator id="copy data account_user to org_user"
t=2018-07-02T10:24:11+0000 lvl=info msg="Skipping migration condition not fulfilled" logger=migrator id="copy data account_user to org_user"
t=2018-07-02T10:24:11+0000 lvl=info msg="Starting plugin search" logger=plugins
t=2018-07-02T10:24:11+0000 lvl=info msg="Initializing Alerting" logger=alerting.engine
t=2018-07-02T10:24:11+0000 lvl=info msg="Initializing CleanUpService" logger=cleanup
t=2018-07-02T10:24:11+0000 lvl=info msg="Initializing Stream Manager"
t=2018-07-02T10:24:11+0000 lvl=info msg="Initializing HTTP Server" logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
...
t=2018-07-02T10:26:03+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=37 name="Replicas alert - test" error="Could not find datasource Data source not found" changing state to=alerting
...

[kaazoo]

The problem seems to be caused by grafana-watcher:
https://github.com/coreos/prometheus-operator/blob/7488886938ec366e96c8e66dd8a4e66f0dffb825/contrib/grafana-watcher/updater/dashboard.go#L51

It deletes all existing dashboards before deploying the default dashboards from the asset folder again.

[camilb]

Yes, I am aware of this issue. Grafana config reloader was deprecated and I plan to remove it an use the new Grafana functionality to achieve this. Ref: #95