Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simple caching for kpy bash #42

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

akhanf
Copy link
Contributor

@akhanf akhanf commented Aug 21, 2023

If the cache doesn't exist (or if reset flag is used), a kpy_cache.sh script is created that initializes KSLURM_COMPUTE_NODES and PIP_WHEEL_DIR. I use another file to store the list of nodes since it was quite long to encode inline.

Kept everything else identical (just moved to the cache generation).

If the cache doesn't exist (or if reset flag is used), a kpy_cache.sh script is created that initializes KSLURM_COMPUTE_NODES and PIP_WHEEL_DIR. I use another file to store the list of nodes since it was quite long to encode inline.
@pvandyken
Copy link
Owner

The reason I never cached initially was because I was concerned about graham updating their node list, and then we wouldn't be aware of nodes without internet. I wonder if a reasonable compromise is to recompute the cache once a week or so (i.e. if the cache file is older than a week old)

@akhanf
Copy link
Contributor Author

akhanf commented Aug 22, 2023

Yes I think re-generating the cache if it is over 7 days old is good enough! I'll try that out (just looking at file modification times) once graham starts behaving again..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants