Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up update-python3-dependencies #7234

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Speed up update-python3-dependencies #7234

wants to merge 1 commit into from

Conversation

legoktm
Copy link
Member

@legoktm legoktm commented Sep 23, 2024

Status

Ready for review

Description of Changes

The make update-python3-dependencies step is slow for two main reasons: container image layering and pip-compile.

Currently, each pip-compile step is run in a separate dev-shell, which means after each one, if anything changed in the requirements/ folder, it needs to rebuild the slim image and reinstall all the pip dependencies.

Now I've moved the steps to a separate script that execute in a single dev-shell, so no image rebuilding happens during the updates, it'll just be needed once afterwards.

For an additional boost, switch to the new uv tool, which reimplements pip-compile in a much faster way. The output is basically the same, except the sorting is smarter (e.g. pytest comes before pytest-cov) and package names are properly normalized. We can also drop the --allow-unsafe because uv is entirely independent of setuptools and pip-tools.

uv is still quite new to the Python ecosystem, but this allows us to begin using it without any lock-in, it should be trivial to swap back to pip-tools if needed.

Overall make update-python3-dependencies now takes seconds to run instead of minutes \o/

Refs freedomofpress/securedrop-tooling#16.

Testing

How should the reviewer test this PR?

  • Visual review; no changes to dependencies outside of removal of pip-tools
  • Run make update-python3-dependencies yourself.

Deployment

Any special considerations for deployment? n/a

Checklist

  • Linting (make lint) and tests (make test) pass in the development container
  • Linting and tests (make -C admin test) pass in the admin development container
  • I have written a test plan and validated it for this PR
  • I have opened a PR in the docs repo for these changes, or will do so later
    • I'll do this post-merge

@legoktm legoktm requested a review from a team as a code owner September 23, 2024 20:12
@legoktm
Copy link
Member Author

legoktm commented Sep 23, 2024

This is a rather conservative PR, there are other places we could be swapping in uv for speed like in CI or in admin/, but I think we can start with this for now and expand it over time.

The `make update-python3-dependencies` step is slow for two main
reasons: container image layering and pip-compile.

Currently, each `pip-compile` step is run in a separate dev-shell, which
means after each one, if anything changed in the requirements/ folder,
it needs to rebuild the slim image and reinstall all the pip
dependencies.

Now I've moved the steps to a separate script that execute in a single
dev-shell, so no image rebuilding happens during the updates, it'll just
be needed once afterwards.

For an additional boost, switch to the new uv tool, which reimplements
pip-compile in a much faster way. The output is basically the same,
except the sorting is smarter (e.g. pytest comes before pytest-cov) and
package names are properly normalized. We can also drop the
`--allow-unsafe` because uv is entirely independent of setuptools and
pip-tools.

uv is still quite new to the Python ecosystem, but this allows us to
begin using it without any lock-in, it should be trivial to swap back to
pip-tools if needed.

Overall `make update-python3-dependencies` now takes seconds to run
instead of minutes \o/

Refs <freedomofpress/securedrop-tooling#16>.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready to go
Development

Successfully merging this pull request may close these issues.

1 participant