Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor data step, inference step, Jupyter notebooks #97

Merged
merged 114 commits into from
Dec 11, 2023
Merged
Show file tree
Hide file tree
Changes from 112 commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
4d56047
data step: refactor
raehik Aug 31, 2023
21e0add
use new code paths in training step; fix MLproject
raehik Aug 31, 2023
8cd24a4
data: enable selecting Pangeo intake catalog
raehik Aug 31, 2023
8f35643
MLproject: main->data, rename project
raehik Sep 1, 2023
f9dec0a
tweak MLproject, readme, data step CLI help
raehik Sep 1, 2023
fdb9fa0
cli/data: re-add logging
raehik Sep 1, 2023
5f9dab7
cli/data: +log bounding operation
raehik Sep 11, 2023
0748393
step/data: simplify gaussian_filter call
raehik Sep 11, 2023
0346a4e
cli/data: fix forcing compute call arg order
raehik Sep 11, 2023
c46a6e0
step/data: remove code copied from unused debug
raehik Sep 13, 2023
eb8586a
step/data: fix coarsening scale args
raehik Sep 13, 2023
ab1880d
step/data/coarsen: cleaning
raehik Sep 14, 2023
228ca93
step/data: simplify ufunc call
raehik Sep 14, 2023
30e10c5
readme: clarify data step CLI
raehik Sep 14, 2023
5dbc603
step/data: clean up, log more
raehik Sep 20, 2023
92ee219
bounding box: add validate function
raehik Sep 20, 2023
3cfb298
step/data: clean up some to-dos
raehik Sep 20, 2023
bff9989
fix data step pytests
raehik Sep 20, 2023
ca49b1c
MLproject: remove data step
raehik Sep 22, 2023
53f1263
readme: fix example commands
raehik Sep 22, 2023
08967b7
readme: +note on --help for data step CLI
raehik Sep 22, 2023
4a022d1
cli/data: fix bound_dataset call
raehik Sep 22, 2023
16158b7
readme: +note on old GEOS req
raehik Sep 22, 2023
cd85ea9
rework figure 1 generator notebook
raehik Oct 17, 2023
c0e2727
cli-configs: add figure configs
raehik Oct 18, 2023
1e61763
fix figure notebook configs
raehik Oct 18, 2023
0d9e5e8
rework figure 6 generator notebook
raehik Oct 18, 2023
23e3a76
Jupyter notebooks/global ctrl: begin rework
raehik Oct 26, 2023
616e4f2
begin new inference CLI script
raehik Oct 26, 2023
d778711
pyproject: clean up dependencies
raehik Nov 1, 2023
f8c3fd7
various inference step refactoring
raehik Nov 1, 2023
3fc003e
CI/linting: install required Python type stub libs
raehik Nov 2, 2023
e876ec7
analysis/multiscale: fix imports
raehik Nov 2, 2023
7a24171
analysis/analysis: move global state to save_fig()
raehik Nov 2, 2023
231156c
add some function type annotations
raehik Nov 2, 2023
4bd7dcd
readme: clean up intro
raehik Nov 7, 2023
59694fe
src: various inference, CLI changes
raehik Nov 7, 2023
cc883dd
train/base: clarify data loading comments
raehik Nov 7, 2023
421bb57
docs/data: link to CM2.6 dataset
raehik Nov 7, 2023
1f74b43
cli/inference-test: +describe points about input, model
raehik Nov 8, 2023
9959006
readme: various fixes, changes
raehik Nov 8, 2023
cfe52f8
MLproject/train: improve default test_split
raehik Nov 9, 2023
9ab746c
MLproject/train: revert
raehik Nov 9, 2023
a1c52b7
readme: update training step docs
raehik Nov 9, 2023
a3c7ee1
whitespace nit
raehik Nov 9, 2023
c5901a1
tidy up start of test_global_control notebook
dorchard Nov 9, 2023
1db29de
notebooks/readme: +note on common deps
raehik Nov 9, 2023
fcd6dea
readme: start new training step cmd example
raehik Nov 9, 2023
0d70488
various readme, docs updates
raehik Nov 10, 2023
1887b39
examples/cli: +quick data run config
raehik Nov 10, 2023
89ecf35
move trainScript -> cli/train
raehik Nov 10, 2023
499f181
step/data: rename vars to clarify dataflow
raehik Nov 10, 2023
95b48b3
add example Slurm jobs
raehik Nov 10, 2023
f5c4667
readme: clean up
raehik Nov 10, 2023
2a33a3b
notebooks/readme: +note on common deps
raehik Nov 10, 2023
4b63e62
cli/train: fix some bad CLI option defaults
raehik Nov 10, 2023
01af5cc
readme: tweak
raehik Nov 10, 2023
de16954
notebooks: more work on test-global
raehik Nov 10, 2023
f43afe4
cli: more inference work, cleanup
raehik Nov 10, 2023
9edc753
notebooks: cleanup; prefix paper notebooks
raehik Nov 10, 2023
3c33204
begin overhauling train step
raehik Nov 13, 2023
b535458
cli/train: executes (but freezes)
raehik Nov 13, 2023
e3161a1
cli/train: fix default device
raehik Nov 14, 2023
1a54864
step/data/lib: fix erroneous variable usage
raehik Nov 14, 2023
e664357
readme: update; remove mlflow notes
raehik Nov 17, 2023
3a51f52
various training code fixes
raehik Nov 17, 2023
415f628
datasets/ConcatDataset_: always enforce same dims
raehik Nov 17, 2023
9138fe4
cli/train: simplify to get running
raehik Nov 21, 2023
4e705e9
readme: simplify data, training notes
raehik Nov 21, 2023
fa7eaa7
tinker
raehik Nov 21, 2023
29efba2
examples/cfg/data-paper: generalize
raehik Nov 28, 2023
bb50ab2
exaples/cfg: tweak
raehik Nov 28, 2023
bd8772b
cli/infer: initial working script
raehik Nov 28, 2023
22c097b
cli/train: fiddle with dataset splitting
raehik Nov 28, 2023
a2f567b
cli/infer: fix passed channel names
raehik Nov 29, 2023
58a7150
inference/utils: +arg doc, todo note
raehik Nov 29, 2023
ecfc72f
fix last paper Jupyter notebook
raehik Nov 29, 2023
12f9b70
tweak a Jupyter notebook
raehik Nov 29, 2023
e4ff8b9
step/data/lib: scale is float, not int
raehik Nov 29, 2023
74fba9e
cli/infer: remove irrelevant line
raehik Nov 30, 2023
de5dd33
various cleanup
raehik Nov 30, 2023
b918201
delete old wip scripts
raehik Nov 30, 2023
42dbee9
clean up infer script, lib
raehik Nov 30, 2023
f2f8dcc
add quick note on paper reproduction steps
raehik Nov 30, 2023
ca24253
tweak readme
raehik Nov 30, 2023
d3d36df
cli: tweak
raehik Dec 1, 2023
b30bac7
tweak readme: examples->resources
raehik Dec 4, 2023
fb67d46
notebook/fig-457: save both data & pred versions of one figure
raehik Dec 4, 2023
a7da168
notebooks: gitignore figures
raehik Dec 4, 2023
6d693ce
notebooks: clean up docs
raehik Dec 4, 2023
87a1664
readme: scrap pointing to data step lib functions
raehik Dec 4, 2023
67c19dd
add some project/repo admin notes
raehik Dec 4, 2023
776c2bc
docs/admin: +few more notes
raehik Dec 5, 2023
a425316
clean up CLI configs
raehik Dec 5, 2023
895fb93
clean up training code, configs
raehik Dec 5, 2023
764d23d
cleanup
raehik Dec 5, 2023
6d5d651
cli/data: match previous code closer
raehik Dec 5, 2023
0582c9f
update Nix flake
raehik Dec 6, 2023
caf9a7d
forcing generation: return to map_blocks
raehik Dec 6, 2023
7bddf76
tests: fix import error
raehik Dec 6, 2023
5095ce7
cli/data: allow specifying Dask num_workers
raehik Dec 6, 2023
5a26e1b
flag _cli_desc with comments pointing out that this describes the module
dorchard Dec 6, 2023
71f1fe9
additional comment
dorchard Dec 6, 2023
f3279c9
notebooks: tweak docs
raehik Dec 8, 2023
63900af
notebooks: tweak paths
raehik Dec 8, 2023
99baa8d
cli/train: clean up
raehik Dec 8, 2023
301f1bb
cli/test: re-add (WIP)
raehik Dec 8, 2023
fe047da
readme: +note on inference script
raehik Dec 8, 2023
a2dcdfd
readme: +note on Hugging Face data res
raehik Dec 8, 2023
8b4a3ba
docs/2021-paper-repro: describe training, predicting
raehik Dec 8, 2023
25c834c
resources/cli-configs/train-paper: note where from
raehik Dec 8, 2023
68b4151
train/base: remove debug prints
raehik Dec 8, 2023
56b2ee0
lib/data: clarify rechunking comment
raehik Dec 8, 2023
e426891
lib/data: remove commented-out temperature interp
raehik Dec 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/linting.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ jobs:

- run: pip install mypy

- name: Install relevant Python type stub libraries
run: pip install types-requests

- name: mypy
run: mypy --strict .

Expand Down
18 changes: 4 additions & 14 deletions MLproject
Original file line number Diff line number Diff line change
@@ -1,20 +1,8 @@
name: cmip2_6
# vim: ft=yaml

# conda_env: conda.yaml
name: gz21_ocean_momentum

entry_points:
main:
parameters:
ntimes : {type: float, default: 10000}
CO2: {type: float, default: 0}
lat_min : float
lat_max : float
long_min : float
long_max : float
factor: {type: float, default: 0}
chunk_size: {type: string, default: 50}
global: {type: str, default: 0}
command: "python src/gz21_ocean_momentum/cmip26.py {lat_min} {lat_max} {long_min} {long_max} --CO2 {CO2} --ntimes {ntimes} --factor {factor} --chunk_size {chunk_size} --global_ {global}"

train:
parameters:
Expand All @@ -35,9 +23,11 @@ entry_points:
submodel : {type: string, default : transform3}
features_transform_cls_name : {type : string, default : None}
targets_transform_cls_name : {type : string, default : None}
subdomains_file: {type: string, default: resources/cli-configs/train-subdomains-paper.yaml}
command: "python src/gz21_ocean_momentum/trainScript.py
--run-id {run_id}
--forcing-data-path {forcing_data_path}
--subdomains-file {subdomains_file}
--batchsize {batchsize} --learning_rate {learning_rate}
--n_epochs {n_epochs} --train_split {train_split}
--test_split {test_split} --time_indices {time_indices}
Expand Down
Loading
Loading