Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimisations #8

Merged
merged 4 commits into from
Nov 18, 2024
Merged

Optimisations #8

merged 4 commits into from
Nov 18, 2024

Conversation

mbway
Copy link
Collaborator

@mbway mbway commented Nov 17, 2024

This PR introduces some optimisations to improve import times (and therefore startup times).

Initially in #7 I requested that the cache is able to be disabled entirely, but with importlib.reload() being an edge case and importlib.invalidate_caches() existing it seems sufficient that the maturin import hook supports this way of clearing the cache to ensure new modules are detected.

Benchmarking

Benchmarking was carried out using the tests/create_benchmark_data.py script. this script creates a scenario close to the worst case, with 100 paths in sys.path each very long nested paths with two pyproject.tomls present on the path back to the root of the filesystem that must be checked.

Results:

These results are run on the slowest storage I have (a USB drive). Each value is the average over 3 runs.
The 'cold cache' runs are after running echo 3 | sudo tee /proc/sys/vm/drop_caches

cold cache warm cache
main 8.80s 6.77s
is_maybe_maturin_project optimisation 7.36s 5.53s
lru_cache 1.95s 0.62s
find_dist_info_path optimisation 1.69s 0.53s

(note that for a more reasonable case such as 20 paths in sys.path, the time taken is ~0.01s)

@mbway mbway assigned messense and unassigned messense Nov 17, 2024
@mbway mbway requested a review from messense November 17, 2024 16:19
@mbway mbway mentioned this pull request Nov 17, 2024
@mbway mbway self-assigned this Nov 17, 2024
@messense messense merged commit 49f973b into PyO3:main Nov 18, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants