Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH: Updates to PANGAEA collection and cleaning pipeline #55

Merged
merged 92 commits into from
May 8, 2024
Merged

ENH: Updates to PANGAEA collection and cleaning pipeline #55

merged 92 commits into from
May 8, 2024

Conversation

scottclowe
Copy link
Member

No description provided.

scottclowe added 30 commits March 6, 2023 23:38
@scottclowe
MNT: Show which ds has None for size
9a64b6e
@scottclowe
ENH: Add option to control whether URL columns are required
0460e60
@scottclowe
API: Disable ensure url option in search
b038c4f
@scottclowe
MNT: Handle alt campaign name within set_metadata
09006e2
@scottclowe
MNT: Use dataset ID as alt instead of DOI
ac76bc7
@scottclowe
MNT: Skip children of parents without URL
f156f49
@scottclowe
STY: Split arg per line with ensure_url included
d8894f9
@scottclowe
ENH: Get lat/lon from PanDataSet if not scraped
1392028
@scottclowe
ENH: Add auth_token support
f52e3b5
@scottclowe
MNT: Only print saving dataframe if verbosity high enough
4da4e30
@scottclowe
BUG: Remove unused import of IPython, not specified in requirements
aec74c6
@scottclowe
BUG: Skip non-CSV files when processing outputs
682fba4
@scottclowe
MNT: Don't add dummy campaign and site columns when downloading datasets
759b1be
@scottclowe
MNT: Print message explaining errors being repeated
18651b8
@scottclowe
MNT: Ignore existing dataset and site columns
5332481
@scottclowe
MNT: Change default site to be based on dataset name, not DOI
df49254
@scottclowe
BUG: Reflect latitudesouth, latitude-, longitudewest, longitude-
a578655
@scottclowe
MNT: Save results for parents whose children don't have URLs
ea5eba4
@scottclowe
MNT: Inherit verbosity from caller
8203880
@scottclowe
MNT: Read CSV files without low_memory mode due to 'mixed types'
77d4cf1
@scottclowe
ENH: Interpolate or extract missing lat, lon, datetime metadata
db0fae0
@scottclowe
MNT: Increase default verbosity level 0->1
2c16f24
@scottclowe
MNT: Exclude some more dataset titles
073efff
@scottclowe
MNT: Manually exclude dataset 805690, which was downloaded without it…
0a0b632 
…s title?
@scottclowe
MNT: Make final report only appear if verbosity enabled
2bbbd66
@scottclowe
MNT: Remove unused import of dateutil.parser
37bedd6
@scottclowe
ENH: Use caching functionality built into PanDataSet
f7a44ee
@scottclowe
MNT: Extract datetime from filename for rest of 896160 series
1643a8b
@scottclowe
MNT: Extract from filename from two more datasets
5e4cf23
@scottclowe
MNT: Also manually exclude 803979, parent of 805690
2f0d49c
scottclowe added 27 commits March 30, 2023 23:32
@scottclowe
MNT: Rearrange so old columns are dropped before mapping new ones ont…
eda209d 
…o them
@scottclowe
MNT: Change warning colour from red to yellow
4d57188
@scottclowe
MNT: Change Campaign -> campaign
0dc20c6
@scottclowe
BUG: Add pangaea- to ds_id for dataframe output
87452af
@scottclowe
MNT: Allow photographs of tiles
d135600
@scottclowe
ENH: Include parent_ds_id in output dataframe
a3b14e2
@scottclowe
MNT: Remove self-imposed rate-limit so cached data is loaded immediately
2082aa3
@scottclowe
ENH: Include url_thumbnail column
f236e0c
@scottclowe
ENH: Find area columns encoding image area in square meters
2873911
@scottclowe
MNT: Find and remove additional FAVOURITE duplicate images
6a5c687
@scottclowe
MNT: Print files which had duplicated URLs resolved
2808ad9
@scottclowe
Revert "MNT: Remove self-imposed rate-limit so cached data is loaded …
aadf712 
…immediately"

This reverts commit 2082aa3.
@scottclowe
MNT: Print number of records before and after dropping duplicates
4f152f1
@scottclowe
MNT: Print IDs of datasets which may have label columns
df0beea
@scottclowe
MNT: Rename parent_ds_id -> collection
f9d5b48
@scottclowe
BUG: Fix nanosecond output format of datetime in pangaea-907025
45a3492
@scottclowe
MNT: Convert parent_ds_id into pangaea-IDENTIFIER like ds_id
13080a1
@scottclowe
DOC: Fix rate limit comment
1db2a4a
@scottclowe
ENH: Merge down metadata across rows with repeated URLs, preserving d…
b1b2a70 
…etails
@scottclowe
MNT: Save a copy with duplicates before removing them, so duplicates …
f631dd8 
…can be resolved without merging again
@scottclowe
BUG: Need to convert datetime to string before merging (some are date…
eebb878 
…time objects)
@scottclowe
MNT: Rewrite any(list comp) as any(generator) instead (flake8:C419)
f1cf985 
C419 Unnecessary list comprehension passed to any() prevents short-circuiting - rewrite as a generator.
@scottclowe
MNT: Rename depth columns
4a42f49
@scottclowe
MNT: Exclude AntGlassSponges with DOWN in their URL - not Benthic ima…
e5d9db0 
…gery
@scottclowe
MNT: Skip missing URL cols
3dfaba1
@scottclowe
ENH: Add process_single to cleanup metadata for a single dataset
a8967dc
@scottclowe
JNB: More EDA and new output files
c4ce7ab
@scottclowe scottclowe changed the title ENH: Updates to PANGAEA cleaning pipeline ENH: Updates to PANGAEA collection and cleaning pipeline May 8, 2024
@scottclowe
DEV: Remove malfunctioning pretty-format-json
d8e0aef 
On the GitHub workflow, it is trying to fix the format of the
jupyter notebooks.
@scottclowe scottclowe merged commit e549443 into master May 8, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@scottclowe