Restructure from scripts

[jatkinson1000]

The code is currently run using scripts as main entry points (cmip26.py and trainScript.py).

This is not ideal for distributing as they are long scripts that a user needs to run and change details ('magic numbers') in.
These will also not work in the proposed src/ layout.

Scripts should be broken up in to functions with simple entry/call in main package, and grouped with other relvant code.

However, there is a larger discussion that needs to be held to decide how to break up the code.

• Separate libraries within this repo for data acquisition/processing and model?

• Separate repositories for data acquisition/processing and model?

  • This starts to feel verbose, but is what is required for Huggingface?

• cmip26.py probably needs to be grouped with data acquisition and processing src/gz_ocean_momentum/data/

• trainScript.py probably needs to be grouped with src/gz_ocean_momentum/train/

Related:

• Remove MLFlow framework #4

[raehik]

This is the next large refactoring that we should look at. Mostly a question of how we configure runs, and optionally MLflow. (The training stage in particular I think depends on MLflow to locate data to process. The data processing stage cmip26.py I was able to run without MLflow.)

[raehik]

#85 addresses this for the data step.

[MarionBWeinzierl]

#95 does the training step refactor

[raehik]

We're closing this issue as partially done:

• The data step was refactored in refactor data step script into library (API) and consumer (CLI) #85 (will be properly merged in Refactor data step, inference step, Jupyter notebooks #97), and defines a bunch of library functions which the CLI then calls. Much less top-level code.
• The training step was improved in refactor data step script into library (API) and consumer (CLI) #85 / Refactor data step, inference step, Jupyter notebooks #97 , with a lot of hardcoded values removed. We didn't perform a larger refactor.
• We're not splitting up the code into separate repositories. Lots of coupling remains between steps, and it would be a bit complicated to untangle this (and would mean lots more maintenance stress).
• We did general file renamings (again refactor data step script into library (API) and consumer (CLI) #85 , Refactor data step, inference step, Jupyter notebooks #97).