Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS clusters: get allowVolumeExpansion: true by default in StorageClass, or document procedure to update it once #2906

Closed
consideRatio opened this issue Aug 1, 2023 · 3 comments · Fixed by #4881
Labels
allocation:internal-eng nominated-to-be-resolved-during-q4-2023 Nomination to be resolved during q4 goal of reducing the technical debt

Comments

@consideRatio
Copy link
Contributor

consideRatio commented Aug 1, 2023

We want our StorageClass resource to have allowVolumeExpansion: true, as that makes us able to increase the PVC's requested storage. This is a requirement for the troubleshooting docs here for example.

I believe its not the default at least for AWS based clusters default StorageClass resource, but I think its the default for GCP based clusters.

@yuvipanda
Copy link
Member

A lot of our prometheuses were down, so I did this now.

  1. Make a script:
    #!/bin/bash
    echo "kubectl patch storageclass gp2 --patch '{\"allowVolumeExpansion\": true}'" | deployer use-cluster-credentials $1
  2. Run this against all the clusters. This will just error out in GKE and Azure clusters, and that's fine:
    ls config/clusters | xargs -L1  ./wat.bash

@yuvipanda
Copy link
Member

After that, I re-ran https://github.com/2i2c-org/infrastructure/actions/runs/5781619537 again, which completely passed this time. However, dynamic resizing only takes effect when the pod itself is recreated, so am doing that as well now.

yuvipanda added a commit to yuvipanda/pilot-hubs that referenced this issue Aug 7, 2023
We add this unconditionally to all clusters for simplification,
so we can set storageClass: gp3 for new clusters that come up on
AWS without issue. This doesn't change the default, and does not
change the storageclass in existing clusters. In addition to using
gp3, it also sets reclaimPolicy to Retain, so if the PVC is deleted,
it does not delete the PV or the underlying EBS volume.

Ref 2i2c-org#2906
Ref 2i2c-org#2717
@yuvipanda
Copy link
Member

Given we've already manually done this for existing clusters, I think for new clusters, we should use #2941 and manage storageclass explicitly. This alleviates r2r concerns that @sgibson91 raised in #2718 as well as allows us to use the cheaper & better gp3 storageclass.

@damianavila damianavila moved this to In Progress ⚡ in Sprint Board Aug 7, 2023
@damianavila damianavila moved this from Needs Shaping / Refinement to In progress in DEPRECATED Engineering and Product Backlog Aug 7, 2023
@damianavila damianavila moved this from In Progress ⚡ to Waiting 🕛 in Sprint Board Aug 9, 2023
@yuvipanda yuvipanda removed their assignment Oct 16, 2023
@yuvipanda yuvipanda added the nominated-to-be-resolved-during-q4-2023 Nomination to be resolved during q4 goal of reducing the technical debt label Oct 16, 2023
@damianavila damianavila moved this from Waiting 🕛 to Todo 👍 in Sprint Board Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
allocation:internal-eng nominated-to-be-resolved-during-q4-2023 Nomination to be resolved during q4 goal of reducing the technical debt
Projects
No open projects
Status: Committed 👍
Development

Successfully merging a pull request may close this issue.

2 participants