Migrate from poetry to uv and update packages

[MBARIMike]

This project is stuck at python = "3.10.*" which goes end of life in October 2026. There are some complex dependencies that should be resolved through testing of the notebooks. It'd be nice to update to python = ">=3.12", which is end of life in October 2028.

[MBARIMike]

Thanks @andermi for the offer in #17 (comment). I'm going to try to use the new package management tools to get as much as possible updated. I'll let you know if I'd like to see your branch.

[MBARIMike]

This project has many architecture specific binary dependencies, e.g. for numpy, scipy, netCDF4, and llvm. On a Mac these are satisfied by installing Mac Ports. I've upgraded my Mac to Sequoia 15.5 and needed to do a sudo port migrate -v and I needed to leave the computer running it overnight for it to finish. Just thought I'd make that note here.

[MBARIMike]

Rather that trying to convert the existing pyproject.toml to one that uv can use I decided to declare bankruptcy and start with a uv managed project from scratch. Here's how...

Mac ports installs recent versions of Python. On my system I changed the python & python3 symbolic links in /opt/local/bin to point to python3.13. I then created a stub pyproject.toml file with:

cd /tmp && uv init auv-python

and copied it over the existing one in auv-python with:

export AUV_PYTHON_HOME=~/GitHub/auv-python  # Use the location where you cloned this repo
cp -i /tmp/auv-python/pyproject.toml $AUV_PYTHON_HOME

Then started adding packages to it:

cd $AUV_PYTHON_HOME
uv add --dev pytest
uv add hvplot
uv add datashader

The hvplot and datashader packages have many dependencies that auv-python needs, so these are a good place to start. To add the remaining required packages I iterated on these steps:

1. Run the tests: uv run pytest
2. See what breaks because of a missing module and run: uv add <missing_module>
3. Goto step 1

A goal is to have a minimal pyproject.toml file and if we need a special module that includes dependencies that we also need we can add that module. An example is cmocean which requires matplotlib. So rather than adding matplotlib I add cmocean and get matplotlib for free.

[MBARIMike]

Iterating with uv run pytest and git add <missing_module> is going pretty well. This is the progress so far:

➜  auv-python git:(uv-update) ✗ uv run pytest
============================================================================================== test session starts ==============================================================================================
platform darwin -- Python 3.13.3, pytest-8.4.0, pluggy-1.6.0
rootdir: /Users/mccann/GitHub/auv-python
configfile: pyproject.toml
collected 5 items

src/data/test_calibrate_align.py .                                                                                                                                                                        [ 20%]
src/data/test_ctd_proc.py ..                                                                                                                                                                              [ 60%]
src/data/test_process_dorado.py F                                                                                                                                                                         [ 80%]
src/data/test_process_i2map.py F

[MBARIMike]

Linting and pytest tests are passing now in CI. Upgraded to Python 3.12. Python 3.13 had problems debugging in VS Code. Dealt with almost all warnings. Still need to rewrite the README.md now that we are using uv. The remaining warning is this one:

auv-python/src/data/calibrate.py:50: UserWarning: The seawater library is deprecated! Please use gsw instead.

[MBARIMike]

Doing a sanity check processing a mission shows good agreement, at least for ctd1_salinity shown here.

Previously processed mission:

Processed with uv-update branch:

[MBARIMike]

Okay. All Notebooks have been executed with the new uv installed modules and resaved with updated known_hashes. As noted in some of the commits some of the interactive plotting features (data value panel sliders) are not working. We can fix those later. This looks ready to merge and use/test in production.

[MBARIMike]

Now testing in production with a reprocessing starting in 2016:

docker compose run --rm auvpython src/data/process_dorado.py --start_year 2016 --end_year 2016 --num_cores 1 -v --noinput --clobber

I use --num_cores 1 because if I'm not in a hurry I like to see the missions processed sequentially – easier to keep track of.

[MBARIMike]

This work has been completed. Now testing a load of 2016-2025 reprocessed data in stoqs_all_doraro_t.