Fixed #495 Reduce Import Time (Lazy Imports)

[seanlaw]

See #495

Pull Request Checklist

Below is a simple checklist but please do not hesitate to ask for assistance!

• Fork, clone, and checkout the newest version of the code
• Create a new branch
• Make necessary code changes
• Install black (i.e., python -m pip install black or conda install -c conda-forge black)
• Install flake8 (i.e., python -m pip install flake8 or conda install -c conda-forge flake8)
• Install pytest-cov (i.e., python -m pip install pytest-cov or conda install -c conda-forge pytest-cov)
• Run black --exclude=".*\.ipynb" --extend-exclude=".venv" --diff ./ in the root stumpy directory
• Run flake8 --extend-exclude=.venv ./ in the root stumpy directory
• Run ./setup.sh dev && ./test.sh in the root stumpy directory
• Reference a Github issue (and create one if one doesn't already exist)

[gitnotebooks]

Review these changes at https://app.gitnotebooks.com/stumpy-dev/stumpy/pull/1122

[seanlaw]

@NimaSarajpoor Please review at your earliest convenience. Apologies for changes in so many files! I realized how inconsistently the unit tests were importing modules (i.e., core, config) vs functions/classes (i.e., stump, mpdist) so I cleaned things up.

Originally, I had noticed that the import time was extremely high immediately after stumpy was installed (~20-30 seconds). With this PR, import time is consistently reduced to 1-2 seconds.

[NimaSarajpoor]

Originally, I had noticed that the import time was extremely high immediately after stumpy was installed (~20-30 seconds). With this PR, import time is consistently reduced to 1-2 seconds

Great improvement!!

Please review at your earliest convenience

Will review it within the next couple of days.

[seanlaw]

I should note that I noticed an issue with lazy imports (this PR) that didn't exist with eager imports (in main)...

Case One:

1. Install this PR
2. Navigate to a new directory
3. Open a python repl
4. Execute import stumpy
5. Execute print(stumpy.stump)

This should print "<function stump at 0x127286b90>" and thus showing that it is a function. However

Case Two:

1. Install this PR
2. Navigate to a new directory
3. Open a python repl
4. Execute import stumpy
5. Type stumpy. (notice the "dot")
6. Hit <tab><tab>
7. Delete all of the tect
8. Execute print(stumpy.stump)

This should print "<module 'stumpy.stump' from '/Users/slaw/Git/stumpy-dev.git/stumpy/stump.py'>" and thus showing that it is a module (not a function). So, somehow, tab-completion causes things to ignore lazy loading and possibly revert/fall back to disambiguating to a module.

In the original (main) __init__.py (without lazy loading), the stump module is disambiguated from the stump function because we've explicitly used:

from .stump import stump

and this instantly makes stumpy.stump refer to the stump function and not the stump module. We will need to somehow replicate this in lazy importing.

[seanlaw]

We will need to somehow replicate this in lazy importing.

This has been fixed in the latest commit 2db3a58

Next, I'll need to run the test suite on Google Colab to ensure that CUDA related things aren't broken

[seanlaw]

Next, I'll need to run the test suite on Google Colab to ensure that CUDA related things aren't broken

I can confirm that all tests are passing on Google Colab! The key was making sure to update the pre-installed numba-cuda package:

pr_number = "1122"  # Replace this with your Pull Request (PR) number

!rm -rf stumpy
!git clone https://github.com/stumpy-dev/stumpy
!cd stumpy && git fetch origin pull/$pr_number/head:pr$pr_number
!cd stumpy && git checkout pr$pr_number
!pip install --upgrade numba-cuda
!cd stumpy && ./pip.sh
!cd stumpy && ./setup.sh
!cd stumpy && ./test.sh gpu

Produces:

Cleaning Up
Checking Black Code Formatting
All done! ✨ 🍰 ✨
96 files would be left unchanged.
Checking iSort Import Formatting
Checking Flake8 Style Guide Enforcement
Checking Missing Docstrings
Checking Package Imports
Ray Not Installed
Checking Missing fastmath flags in njit functions
Checking hardcoded fastmath flags in njit functions
Executing GPU Unit Tests Only
Testing Numba JIT CUDA GPU Compiled Functions
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 248 items                                                            

tests/test_core.py ..................................................... [ 21%]
........................................................................ [ 50%]
........................................................................ [ 79%]
...................................................                      [100%]

============================= 248 passed in 44.30s =============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 2 items                                                              

tests/test_gpu_aampdist.py ..                                            [100%]

============================== 2 passed in 2.91s ===============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 6 items                                                              

tests/test_gpu_aamp_ostinato.py ......                                   [100%]

============================== 6 passed in 4.29s ===============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 37 items                                                             

tests/test_gpu_aamp.py .....................................             [100%]

============================== 37 passed in 6.40s ==============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 2 items                                                              

tests/test_gpu_aamp_stimp.py ..                                          [100%]

============================== 2 passed in 3.58s ===============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 4 items                                                              

tests/test_gpu_mpdist.py ....                                            [100%]

============================== 4 passed in 6.02s ===============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 41 items                                                             

tests/test_gpu_ostinato.py .........................................     [100%]

============================= 41 passed in 19.46s ==============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 3 items                                                              

tests/test_gpu_stimp.py ...                                              [100%]

============================== 3 passed in 7.90s ===============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 39 items                                                             

tests/test_gpu_stump.py .......................................          [100%]

============================== 39 passed in 9.37s ==============================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 46 items                                                             

tests/test_non_normalized_decorator.py ................................. [ 71%]
.............                                                            [100%]

======================== 46 passed in 103.69s (0:01:43) ========================
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /content/stumpy
configfile: pytest.ini
plugins: langsmith-0.6.4, anyio-4.12.1, typeguard-4.4.4
collected 5 items                                                              

tests/test_precision.py .....                                            [100%]

============================== 5 passed in 13.82s ==============================
Cleaning Up

[NimaSarajpoor]

@seanlaw
Left a few comments for your considerations. If you notice a comment does not make sense, please let me know!! I tried to understand the logic implemented in __init__.py. Hopefully I did not miss anything...

[seanlaw]

@seanlaw Left a few comments for your considerations. If you notice a comment does not make sense, please let me know!! I tried to understand the logic implemented in __init__.py. Hopefully I did not miss anything...

Thanks @NimaSarajpoor. I've made all of the suggested changes

[NimaSarajpoor]

Thanks for addressing the comments. No more comments from my side. Since you already tested it in Google Colab, all should be good.

Please let me know if you want me to pay extra attention to a particular part. Otherwise, looks good to be merged! Great work!!

[seanlaw]

Thanks for the assist @NimaSarajpoor!