Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

asynchronous registry fetching #16

Merged
merged 6 commits into from
Aug 19, 2024
Merged

asynchronous registry fetching #16

merged 6 commits into from
Aug 19, 2024

Conversation

fntlnz
Copy link
Collaborator

@fntlnz fntlnz commented Aug 14, 2024

This PR introduces a mechanism that wraps the registry implementation to keep a cache of images and images tags in order to avoid asking the registry over and over.

This was a problem when using the server mode in registries with hundreds images and hundreds of tags where effectively to render the home page we need to also ask the registry for tags. That resulted in thousands of requests to the registries that were repeated over and over.

With this strategy, requests are done only after a set refresh interval --refresh-interval while calling serve to tell staticreg how often to refresh its internal cache from the registry. This mechanism sits on top of the mechanism that caches the HTML of the generated web pages allowing the user to prioritize if having fresh data is more important than being fast (and making less requests) and vice versa.

@fntlnz fntlnz self-assigned this Aug 14, 2024
Copy link

@alberto-miranda alberto-miranda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some very minor comments which are probably a matter of personal opinion. Other than that, I see no problems with the PR.

PS: Since this is a new project and is a bit outside my expertise, I would appreciate it in the future if you could increase the number of descriptive comments 🙏. Thanks!

cmd/serve.go Outdated Show resolved Hide resolved
pkg/server/server.go Outdated Show resolved Hide resolved
fntlnz added a commit that referenced this pull request Aug 16, 2024
fntlnz added a commit that referenced this pull request Aug 16, 2024
@fntlnz fntlnz force-pushed the reduce-registry-calls branch from 3d58b00 to cbb7f7f Compare August 16, 2024 10:12
@fntlnz
Copy link
Collaborator Author

fntlnz commented Aug 16, 2024

Thanks for the review @alberto-miranda - I'll try to document more.

I'm keeping this on hold because there are still a few performance issues I need to tackle.

@fntlnz fntlnz force-pushed the reduce-registry-calls branch from f2c60f4 to 4c3a4b7 Compare August 16, 2024 22:16
@fntlnz
Copy link
Collaborator Author

fntlnz commented Aug 16, 2024

This is ready.
Now the internal cache is structured in a way that we don't need to do many calls to tag lists to render the index so it's scalable even on registries with thousands of repos.
Since we changed our way of using it from the initial idea I also removed the generate command, it didn't make sense to port all these changes there.

@fntlnz fntlnz merged commit d8fadb6 into master Aug 19, 2024
1 check passed
@fntlnz fntlnz deleted the reduce-registry-calls branch August 19, 2024 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants