Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(mypy): add mypy check and adjust code for types #439

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

d33bs
Copy link
Member

@d33bs d33bs commented Sep 15, 2024

Description

This PR adds a check through GitHub Actions for mypy in order to make sure Python types align with certain standards within pycytominer. I've also made some adjustments to the code in order to pass mypy checks.

Closes #437

What is the nature of your change?

  • Bug fix (fixes an issue).
  • Enhancement (adds functionality).
  • Breaking change (fix or feature that would cause existing functionality to not work as expected).
  • This change requires a documentation update.

Checklist

Please ensure that all boxes are checked before indicating that a pull request is ready for review.

  • I have read the CONTRIBUTING.md guidelines.
  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.
  • New and existing unit tests pass locally with my changes.
  • I have added tests that prove my fix is effective or that my feature works.
  • I have deleted all non-relevant text in this pull request template.

📚 Documentation preview 📚: https://pycytominer--439.org.readthedocs.build/en/439/

@d33bs d33bs force-pushed the add-mypy-checks branch 3 times, most recently from d517035 to 77d1bb4 Compare September 15, 2024 20:24
@codecov-commenter
Copy link

codecov-commenter commented Sep 15, 2024

Codecov Report

Attention: Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.

Project coverage is 94.73%. Comparing base (c782728) to head (8478dbd).
Report is 31 commits behind head on main.

Files with missing lines Patch % Lines
pycytominer/cyto_utils/collate.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #439   +/-   ##
=======================================
  Coverage   94.72%   94.73%           
=======================================
  Files          56       57    +1     
  Lines        3147     3151    +4     
=======================================
+ Hits         2981     2985    +4     
  Misses        166      166           
Flag Coverage Δ
unittests 94.73% <93.33%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@d33bs d33bs marked this pull request as ready for review September 16, 2024 17:00
Copy link
Member

@gwaybio gwaybio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I trust Ken more to give final approval

Comment on lines +83 to +84
if output_type is None:
output_type = "csv"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels dangerous - perhaps we could throw a warning that None is not supported?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is standard way of checking if a default none argument has been passed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, I should clarify what I mean by "dangerous" - if a user explicitly sets output_type=None and expects certain behavior (e.g., outputting to variable and not writing), they may be surprised that we overwrite it to output_type="csv". Perhaps the solution isn't to throw a warning (maybe add to docstring)? Or, we could elect to do nothing if this is standard expected behavior.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for these comments! I had to make this change in order to integrate aggregate with output properly given its default parameter types. I recognize we could change the defaults one direction or the other (with aggregate or output) but I didn't want to move too much outside of the scope of the changes here in order to get a first pass with mypy. I wasn't sure about rationalizing a default value for aggregate's output_type if it wasn't going to use output. At the same time, I didn't want to change too much of output's default capabilities due to how it integrates with many other functions (could get complex for the scope of this PR).

My feeling here was that if output was used at all that we'd expect to be "outputting" something, and that if None were passed to it for output_type (especially for nested function calls) that it should revert to the default string value for the parameter in output.

@@ -56,6 +56,20 @@ def test_output_default():
result, DATA_DF, check_names=False, check_exact=False, atol=1e-3
)

# test with an output_type of None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

Copy link
Member

@kenibrewer kenibrewer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a surprisingly low number of changes for enabling mypy in a repo. This might be because of the ignore_missing_imports and allow_redefinitions config options. What was the change burden like for turning those options off?

@d33bs
Copy link
Member Author

d33bs commented Sep 18, 2024

Thanks @gwaybio and @kenibrewer for the reviews.

This is a surprisingly low number of changes for enabling mypy in a repo.

I'm not sure, but I wonder if this has to do with the low amount of type hinting alongside compatibility with inferred types. Could be useful to make a more focused effort towards adding type hints throughout.

What was the change burden like for turning those options off?

When I looked here I noticed we no longer needed the allow_redefinitions config option so I've turned this off. ignore_missing_imports results in the following errors / warnings. Generally I turned this on as I felt there might be many dependencies we would have to add and I wasn't certain of the benefit. Maybe this could be an iterative process where we move towards that in another change? I can also see why we might do it here.

pycytominer/__config__.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/output.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/features.py:6: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:6: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:10: error: Skipping analyzing "boto3": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:11: error: Skipping analyzing "botocore": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:13: error: Skipping analyzing "sqlalchemy": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations_cmd.py:2: error: Skipping analyzing "fire": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:9: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/operations/transform.py:10: error: Skipping analyzing "scipy.stats": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:11: error: Skipping analyzing "sklearn.base": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:12: error: Skipping analyzing "sklearn.preprocessing": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/util.py:8: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/load.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/modz.py:2: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/collate.py:75: error: Skipping analyzing "cytominer_database.ingest": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/collate.py:75: error: Skipping analyzing "cytominer_database": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/collate.py:76: error: Skipping analyzing "cytominer_database.munge": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/write_gct.py:12: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/DeepProfiler_processing.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: Hint: "python3 -m pip install pandas-stubs"
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: (or run "mypy --install-types" to install all missing stub packages)
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
pycytominer/aggregate.py:8: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/normalize.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/normalize.py:6: error: Skipping analyzing "sklearn.preprocessing": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/annotate.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cells.py:4: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cells.py:21: error: Skipping analyzing "sqlalchemy": module is installed, but missing library stubs or py.typed marker  [import-untyped]
Found 26 errors in 16 files (checked 31 source files)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add mypy type check linting
4 participants