Post-Mortem: Reviving a broken ESIP-Fed QHub, Lessons Learned

[dharhas]

On Thursday Dask-Gateway broke on the ESIPFed QHub after a minor update to a conda environment (adding xarray). This rendered it impossible to use Dask-Clusters on ESIPFed's QHub install. Removing the offending package from the environment did not fix the problem and on Friday several of us spent a good part of the day trying to diagnose the problem.

The final diagnosis was the latest version of Bokeh (version 2.3.0) broke several packages in the ecosystem and since Dask-Gateway depends on Bokeh for diagnosis plots, it was failing to start. In the meantime, all the things we had tried to do to fix the system while we were poking around made the installation more fragile.

In hindsight, there are a couple of things we could have done differently and some upcoming features in QHub that would have made Friday less painful.

1. At the core, Dask-Gateway broke because an environment update brought in broken packages. Being able to roll-back to an earlier environment would have solved the problem. Unfortunately, earlier environments had been deleted from the conda-store pod for space. New deployments of QHub, trying to recreate the old environments did not succeed because Bokeh was not pinned in any of the environment.yml specs. If we complete and deploy the new version of conda-store. This should no longer be an issue since rolling back to an earlier working environment should be trivial.
2. From our experience Dask-Gateway typically fails because of python dependency issues. We spent a lot of times trying to make sure kubernetes was working correctly and we probably should have looked at the Dask-Gateway error logs earlier since that is what finally gave us the clue that led us to Bokeh. If the main JHub is up and running then Kubernetes is probably ok.
3. Backups, backups, backups. If we finish our backup story and have snapshots of user data and have an easy way to attach these backups to new QHub, it would be much simpler and quick to redeploy a new QHub. QHub can be deployed in 15-30 mins, if we could attach data to it then it becomes a low effort deal to deploy a clean install of the latest QHub.
4. Simplify the yaml needed to deploy QHub. On Friday, we tried to deploy a new QHub in a rush, the yaml has two places for a url and these didn't match (in one we uses qhub.dev and the other used quansight.dev). Because of that simple mistake, we spent a while trying to diagnose why the deployed QHub had a bad SSL cert. We really need to have a simplified yaml that just collects the data needed and then uses a jinja template or similar to populate the version we actually need.

@jkellndorfer @rsignell-usgs @tylerpotts @aktech @costrouc

Please add any comments you have here.

[Adam-D-Lewis]

I'm closing old discussions in a repo cleanup effort. Feel free to re-open if needed.