[Decommission Hub] Carbon Plan

[colliand]

Summary

CS&S forwarded a request sent on 2023-11-30 requesting that the Carbon Plan hub be decommissioned. The message includes text:

We will migrate all our materials off the 2i2c Hub by December 15th. If possible, we would appreciate it if the Hub is shut down on December 18th or 19th, as CarbonPlan will be on winter break December 20th-Jan 2nd, and it would be ideal if we’re available for any questions during the process.

Info

• Community Representative:
• Link to New Hub issue: New Hub: Carbon Plan #291
• Proposed end date: 2023-12-18

Task List

Phase I

• Confirm with Community Representative that the hub is no longer in use and it's safe to decommission
• Confirm if there is any data to migrate from the hub before decommissioning
  • If yes, confirm where the data should be migrated to
    • Confirm a 2i2c Engineer has access to the destination in order to complete the data migration
  • If no, confirm it is ok to delete all the data stored in the user home directories

Phase II - Hub Removal

(These steps are described in more detail in the docs at https://infrastructure.2i2c.org/en/latest/hub-deployment-guide/hubs/other-hub-ops/delete-hub.html)

• Manage existing data (migrate data from the hub or delete it)
• Delete the hub's authentication application on GitHub or CILogon (note CILogon removal requires the hub config in place)
• Remove the appropriate config/clusters/<cluster_name>/<hub_name>.values.yaml files. A complete list of relevant files can be found under the appropriate entry in the associated cluster.yaml file.
• Remove the associated hub entry from the config/clusters/<cluster_name>/cluster.yaml file.
• Remove the hub deployment
  • helm --namespace HUB_NAME delete HUB_NAME
  • kubectl delete namespace HUB_NAME

Phase III - Cluster Removal

This phase is only necessary for single hub clusters.

• Remove the cluster's datasource from the central Grafana with:
  • deployer grafana central-ds remove CLUSTER_NAME
• Run terraform plan -destroy and terraform apply from the appropriate workspace, to destroy the cluster
• Delete the terraform workspace: terraform workspace delete <NAME>
• Remove the associated config/clusters/<cluster_name> directory and all its contents
• Remove the cluster from CI:
  • deploy-hubs.yaml
  • validate-clusters.yaml
• Remove the cluster from the list of grafana datasources at https://grafana.pilot.2i2c.cloud/datasources
• Remove A record from Namecheap account

[damianavila]

Assigned @yuvipanda to process this decommission given he will be around during the requested shutdown time.

[yuvipanda]

I'd also like us to have an 'exit interview' with them, and I'd like to be present for that. Thoughts, @colliand and @jmunroe?

[jmunroe]

Agreed. See https://2i2c.freshdesk.com/a/tickets/1156 for the follow-up with CarbonPlan. Looks like there might be a 30min already scheduled with @colliand for after AGU.

[maxrjones]

@yuvipanda I'd be glad to chat separately if the already scheduled block at 12-12:30pm ET tomorrow doesn't work for you.

Thank you all for your work on the decommissioning. On the "Confirm if there is any data to migrate from the hub before decommissioning ", does this refer to backing up the NFS? I've worked with users to backup their individual home directories, but I'm curious about whether you have recommendations regarding creating a backup of the full file storage system.

[yuvipanda]

@maxrjones I can tar up everyone's homedirs and put it in your homedir. how does that sound?

I'll try to join the meeting if I can!

[maxrjones]

@maxrjones I can tar up everyone's homedirs and put it in your homedir. how does that sound?

This seems like a good idea to me, thank you!

I know this is unlikely, but we had a Hub on GCP that was decommissioned sometime during late summer to early fall 2022. One person who was on leave during that time lost their home dir during that process. Do you happen to have a record of if any backing up happened, or if the storage part of that infrastructure remains?

[yuvipanda]

It was wonderful chatting with you today, @maxrjones! There's now a homes.tar.gz in your own home directory that is a compressed version of everyone's home directories. Can you download that and let me know when you are done, so I can then proceed to decomissioning? I'll decomission this wed-fri so nobody is interrupted.

Unfortunately I don't think anything remains from the GCP time, so we can not retrieve that.

[maxrjones]

It was wonderful chatting with you today, @maxrjones! There's now a homes.tar.gz in your own home directory that is a compressed version of everyone's home directories. Can you download that and let me know when you are done, so I can then proceed to decomissioning? I'll decomission this wed-fri so nobody is interrupted.

Great chatting with you as well! Great, thank you! I successfully downloaded it.

[yuvipanda]

Great, @maxrjones! I'll decomission this sometime in the next few days and update this issue.

[yuvipanda]

@maxrjones I've cleaned up all the resources that we created. I see a couple of nebari related resources, so have not touched them. I'd appreciate if you (or whoever is taking care of the nebari stuff) could take a look to make sure that all the resources (EBS volumes in particular) that are still present are expected to be there, and not leftovers from our cluster!

[maxrjones]

@maxrjones I've cleaned up all the resources that we created. I see a couple of nebari related resources, so have not touched them. I'd appreciate if you (or whoever is taking care of the nebari stuff) could take a look to make sure that all the resources (EBS volumes in particular) that are still present are expected to be there, and not leftovers from our cluster!

Thanks, Yuvi! I'll take a look tomorrow.

[choldgraf]

FYI I think removing CarbonPlan is messing up our global usage dashboard, I've opened an issue here:

• Grafana global dashboards show a broken data source related to CarbonPlan #3547

[maxrjones]

@maxrjones I've cleaned up all the resources that we created. I see a couple of nebari related resources, so have not touched them. I'd appreciate if you (or whoever is taking care of the nebari stuff) could take a look to make sure that all the resources (EBS volumes in particular) that are still present are expected to be there, and not leftovers from our cluster!

I reached out to the Nebari folks about the EBS volumes and confirmed those resources are still present. There's also a handful of older volumes from 2021 named kubernetes-dynamic-pvc-.... These aren't associated with the nebari infrastructure. It looks like the activity on the 400 GB volume ended at the same time as the resource cleanup, so I'm guessing this is the shared filesystem along with other components from 2i2c. Does that sounds right to you? Do you recommend taking a look at any of the other data or backing up anything apart from what we already archived from the Hub home directories?

[yuvipanda]

@maxrjones ah, I'll take a look at those early next week and see what they may be and get back to you!

[yuvipanda]

@maxrjones I took a quick look and these can also all be removed. kubernetes-dynamic-pvc-297b08cc-43b7-4bc3-8f31-3528898571c1 has prometheus usage data from the cluster, which you can safe if you want. Otherwise these are all safe to delete.

[yuvipanda]

@maxrjones just wanted to check in and see if there's anything else we can do to help :)

[yuvipanda]

@maxrjones I'm going to close this one out! Let us know if there's anything more we need to do :) It was great working with you over the last few years! <3

[maxrjones]

@maxrjones I'm going to close this one out! Let us know if there's anything more we need to do :) It was great working with you over the last few years! <3

Thanks @yuvipanda! It was great working with you as well. Super grateful for all that you and your team members bring to the open source community!