Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added more documentation for dealing with big datasets #408

Merged

Conversation

freemansw1
Copy link
Member

Added more details to our documentation around dealing with big datasets, including a new example notebook of running feature detection in parallel.

With @RBhupi from tobathon February 2024

  • Have you followed our guidelines in CONTRIBUTING.md?
  • Have you self-reviewed your code and corrected any misspellings?
  • Have you written documentation that is easy to understand?
  • Have you written descriptive commit messages?
  • Have you added NumPy docstrings for newly added functions?
  • Have you formatted your code using black?
  • If you have introduced a new functionality, have you added adequate unit tests?
  • Have all tests passed in your local clone?
  • If you have introduced a new functionality, have you added an example notebook?
  • Have you kept your pull request small and limited so that it is easy to review?
  • Have the newest changes from this branch been merged?

@freemansw1 freemansw1 added enhancement Addition of new features, or improved functionality of existing features documentation Updates and improvements to documentation and examples labels Feb 16, 2024
Copy link

codecov bot commented Feb 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.91%. Comparing base (57612ec) to head (e85c446).
Report is 115 commits behind head on RC_v1.5.x.

Additional details and impacted files
@@            Coverage Diff             @@
##           RC_v1.5.x     #408   +/-   ##
==========================================
  Coverage      60.91%   60.91%           
==========================================
  Files             23       23           
  Lines           3541     3541           
==========================================
  Hits            2157     2157           
  Misses          1384     1384           
Flag Coverage Δ
unittests 60.91% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Feb 16, 2024

Linting results by Pylint:

Your code has been rated at 8.70/10 (previous run: 8.70/10, +0.00)
The linting score is an indicator that reflects how well your code version follows Pylint’s coding standards and quality metrics with respect to the RC_v1.5.x branch.
A decrease usually indicates your new code does not fully meet style guidelines or has potential errors.

Copy link
Member

@w-k-jones w-k-jones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The only change I would suggest to the notebook is performing tracking to show that this needs to be performed on a single dataframe, but that it can be done without loading all spatial data into memory. The notebook isn't yet displayed in the docs, is that intended? And the failure of the jupyter notebooks CI can be fixed with the inclusion of dask in example_requirements.txt, which has been done in #334

@freemansw1 freemansw1 mentioned this pull request Feb 23, 2024
2 tasks
@freemansw1 freemansw1 changed the base branch from tobathon to RC_v1.5.x March 13, 2024 15:29
@freemansw1 freemansw1 added this to the Version 1.5.3 milestone Mar 13, 2024
@freemansw1 freemansw1 marked this pull request as ready for review May 9, 2024 21:23
@freemansw1 freemansw1 requested a review from JuliaKukulies May 9, 2024 21:24
@freemansw1
Copy link
Member Author

I'm ready for a review!

@w-k-jones w-k-jones self-requested a review May 17, 2024 14:18
@fsenf fsenf requested review from fsenf and removed request for JuliaKukulies May 17, 2024 14:22
Copy link
Member

@w-k-jones w-k-jones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me! Happy for this to be merged when ready

Copy link
Member

@fsenf fsenf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nice addition. Thank you for your work!

I requested some formatting changes - which will be easy to do

doc/big_datasets_examples/index.rst Outdated Show resolved Hide resolved
doc/big_datasets.rst Outdated Show resolved Hide resolved
@freemansw1
Copy link
Member Author

Ok, @fsenf, I believe I've addressed your comments and I am now ready for a re-review.

Copy link
Member

@fsenf fsenf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for update! I am happy with the changes. I especially like that the doc notebooks have been integrated into the example gallery: https://tobac--408.org.readthedocs.build/en/408/examples.html

Please go ahead and merge!

Cheers, Fabian.

PS: It would be cool to have nice looking thumbnails for the new gallery entry. I will create a new issue for that request.

@freemansw1 freemansw1 merged commit 67f86cf into tobac-project:RC_v1.5.x Aug 15, 2024
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Updates and improvements to documentation and examples enhancement Addition of new features, or improved functionality of existing features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants