WIP on combine.py

.vscode/launch.json

@@ -39,13 +39,24 @@
             "args": ["-v", "1", "-d", "0", "-i", "data/auv_data/dorado/missionlogs/2009.055.05/lopc.bin", "-n", "data/auv_data/dorado/missionnetcdfs/2009.055.05/lopc.nc", "-f", "--LargeCopepod_AIcrit", "0.3"]
         },
         {
-            "name": "1.1 - correct_log_times.py --mission 2017.284.00 --auv_name Dorado389",
+            "name": "1.2 - correct_log_times.py --mission 2017.284.00 --auv_name Dorado389",
             "type": "debugpy",
             "request": "launch",
             "program": "${workspaceFolder}/src/data/correct_log_times.py",
             "console": "integratedTerminal",
             "args": ["--auv_name", "Dorado389", "--mission", "2017.284.00", "-v", "2"]
         },
+        {
+            "name": "1.3 - nc42netcdfs",
+            "type": "debugpy",
+            "request": "launch",
+            "program": "${workspaceFolder}/src/data/nc42netcdfs.py",
+            "console": "integratedTerminal",
+            // A small log_file that has a reasonable amount of data, and known_hash to verify download
+            //"args": ["-v", "1", "--log_file", "ahi/missionlogs/2025/20250908_20250912/20250911T201546/202509112015_202509112115.nc4", "--known_hash", "d1235ead55023bea05e9841465d54a45dfab007a283320322e28b84438fb8a85"]
+            // Has bad latitude and longitude values
+            "args": ["-v", "1", "--log_file", "brizo/missionlogs/2025/20250909_20250915/20250914T080941/202509140809_202509150109.nc4"]
+        },
         {
             "name": "2.0 - calibrate.py",
             "type": "debugpy",
@@ -92,6 +103,16 @@
             "program": "${workspaceFolder}/src/data/hs2_proc.py",
             "console": "integratedTerminal",
         },
+
+        {
+            "name": "2.2 - combine.py",
+            "type": "debugpy",
+            "request": "launch",
+            "program": "${workspaceFolder}/src/data/combine.py",
+            "console": "integratedTerminal",
+            "justMyCode": false,
+            "args": ["-v", "1", "--log_file", "brizo/missionlogs/2025/20250909_20250915/20250914T080941/202509140809_202509150109.nc4"]
+        },
         {
             "name": "3.0 - align.py",
             "type": "debugpy",
@@ -282,5 +303,17 @@
             "console": "integratedTerminal",
             "args": ["-v", "1", "--noinput", "--no_cleanup", "--download", "--mission", "2011.256.02"]
         },
+        {
+            "name": "process_lrauv",
+            "type": "debugpy",
+            "request": "launch",
+            "program": "${workspaceFolder}/src/data/process_lrauv.py",
+            "console": "integratedTerminal",
+            //"args": ["-v", "1", "--log_file", "brizo/missionlogs/2025/20250909_20250915/20250914T080941/202509140809_202509150109.nc4"]
+            //"args": ["-v", "2", "--log_file", "brizo/missionlogs/2025/20250909_20250915/20250914T080941/202509140809_202509150109.nc4", "--clobber"]
+            "args": ["-v", "2", "--log_file", "brizo/missionlogs/2025/20250909_20250915/20250914T080941/202509140809_202509150109.nc4", "--clobber", "--no_cleanup"]
+            //"args": ["-v", "1", "--auv_name", "tethys", "--start", "20120901", "--end", "20121101", "--noinput"]
+        },
+
     ]
 }

WORKFLOW.md → DORADO_WORKFLOW.md

@@ -1,6 +1,6 @@
-## Data Workflow
+## Dorado Data Workflow
 
-The sequence of steps to process data is as follows:
+The sequence of steps to process Dorado data is as follows:
 
   logs2netcdfs.py → calibrate.py → align.py → resample.py → archive.py → plot.py
 
@@ -70,6 +70,6 @@ on the local file system's work directory is as follows:
 
     archive.py
         Copy the netCDF files to the archive directory. The archive directory
-        is initally in the AUVCTD share on atlas which is shared with the
+        is initially in the AUVCTD share on atlas which is shared with the
         data from the Dorado Gulper vehicle, but can also be on the M3 share
         on thalassa near the original log data.

LRAUV_WORKFLOW.md

@@ -0,0 +1,68 @@
+## LRAUV Data Workflow
+
+The sequence of steps to process LRAUV data is as follows:
+
+  nc42netcdfs.py → combine.py → align.py → resample.py → archive.py → plot.py
+
+Details of each step are described in the respective scripts and in the
+description of output netCDF files below. The output file directory structure
+on the local file system's work directory is as follows:
+
+    ├── data
+    │   ├── lrauv_data
+    │   │   ├── <auv_name>           <- e.g.: ahi, brizo, pontus, tethys, ...
+    │   │   │   ├── missionlogs/year/dlist_dir
+    │   │   │   │   ├── <log_dir>    <- e.g.: ahi/missionlogs/2025/20250908_20250912/20250911T201546/202509112015_202509112115.nc4
+    │   │   │   │   │   ├── <nc4>    <- .nc4 file containing original data
+    │   │   │   │   │   ├── <nc>     <- .nc files, one for each group from the .nc4 file
+    |   |   |   |   |   |                data identical to original in NETCDF4 format
+    │   │   │   │   │   ├── <_cal>   <- A single NETCDF3 .nc file containing all the
+    |   |   |   |   |   |               varibles from the .nc files along with nudged
+    |   |   |   |   |   |               latitudes and longitudes - created by combine.py
+    │   │   │   │   │   ├── <_align> <- .nc file with all measurement variables
+    |   |   |   |   |   |               having associated coordinate variables
+    |   |   |   |   |   |               at original instrument sampling rate -
+    |   |   |   |   |   |               created by align.py
+    │   │   │   │   │   ├── <_nS>    <- .nc file with all measurement variables
+                                        resampled to a common time grid at n
+                                        Second intervals - created by resample.py
+
+    nc42netcdfs.py
+        Extract the groups and the variables we want from the groups into 
+        individual .nc files. These data are saved using NETCDF4 format as
+        there are many unlimited dimensions that are not allowed in NETCDF3.
+        The data in the .nc files are identical to what is in the .nc4 groups.
+
+    combine.py
+        Apply calibration coefficients to the original data. The calibrated data
+        are written to a new netCDF file in the missionnetcdfs/<mission>
+        directory ending with _cal.nc. This step also includes nudging the
+        underwater portions of the navigation positions to the GPS fixes
+        done at the surface and applying pitch corrections to the sensor
+        depth for those sensors (instruments) for which offset values are
+        specified in SensorInfo. Some minimal QC is done in this step, namely
+        removal on non-monotonic times. The record variables in the netCDF
+        file have only their original coordinates, namely time associated with
+        them.
+
+    align.py
+        Interpolate corrected lat/lon variables to the original sampling
+        intervals for each instrument's record variables. This format is
+        analogous to the .nc4 files produced by the LRAUV unserialize
+        process. These are the best files to use for the highest temporal
+        resolution of the data. Unlike the .nc4 files align.py's output files
+        use a naming convention rather than netCDF4 groups for each instrument.
+
+    resample.py
+        Produce a netCDF file with all of the instrument's record variables
+        resampled to the same temporal interval. The coordinate variables are
+        also resampled to the same temporal interval and named with standard
+        depth, latitude, and longitude names. These are the best files to
+        use for loading data into STOQS and for analyses requiring all the
+        data to be on the same spatial temporal grid.
+
+    archive.py
+        Copy the netCDF files to the archive directory. The archive directory
+        is initially in the AUVCTD share on atlas which is shared with the
+        data from the Dorado Gulper vehicle, but can also be on the M3 share
+        on thalassa near the original log data.

README.md

@@ -60,7 +60,7 @@ print out the usage information for each of the processing scripts:
     uv run src/data/process_i2map.py --help  
     uv run src/data/process_dorado.py --help  
 
-See [WORKFLOW.md](WORKFLOW.md) for more details on the data processing workflow.
+See [DORADO_WORKFLOW.md](DORADO_WORKFLOW.md) for more details on the data processing workflow.
 
 ### Jupyter Notebooks ###
 To run the Jupyter Notebooks, start Jupyter Lab at the command line with:

TROUBLESHOOTING.md

@@ -14,7 +14,7 @@ and make sure that it's the only entry in "process_dorado" that is uncommented.
 
 2. From VS Code's Run and Debug panel select "process_dorado" and click the green Start Debugging play button. For data to be copied from the archive the smb://atlas.shore.mbari.org/AUVCTD share must be mounted on your computer. Primary development is done in MacOS where the local mount point is /Volumes. Archive volumes are hard-coded as literals in [src/data/process_dorado.py](https://github.com/mbari-org/auv-python/blob/fc3b58613761b295ab47907993c4d0eb0bceb197/src/data/process_dorado.py) and [src/data/process_i2map.py](https://github.com/mbari-org/auv-python/blob/fc3b58613761b295ab47907993c4d0eb0bceb197/src/data/process_i2map.py). These should be changed if you mount these volumes at a different location.
 
-3. Mission log data will copied to your `auv-python/data/auv_data/` directory into subdirectories organized by vehicle name, mission, and processing step. Data will be processed as described in [WORKFLOW.md](WORKFLOW.md). A typical mission takes about 10 minutes to process.
+3. Mission log data will copied to your `auv-python/data/auv_data/` directory into subdirectories organized by vehicle name, mission, and processing step. Data will be processed as described in [DORADO_WORKFLOW.md](DORADO_WORKFLOW.md). A typical mission takes about 10 minutes to process.
 
 4. After all of the intermediate files are created any step of the workflow may be executed and debugged in VS Code. The `.vscode\launch.json` file has several example entries that can be modified for specific debugging purposes via the menu in the Run and Debug panel.
 

notebooks/README.md

@@ -1,5 +1,6 @@
 The Notebooks in this directory are intended to be used to examine the data
-generated by each of the steps described in the [workflow]("../WORKFLOW.md"):
+generated by each of the steps described in the [Dorado]]("../DORADO_WORKFLOW.md")
+or [LRAUV]("../LRAUV_WORKFLOW.md") WORKFLOW documents:
 
     logs2netcdfs.py → calibrate.py → align.py → resample.py → archive.py → <ML operations & analysis>
                    1.x            2.x        3.x           4.x          5.x          6.x

pyproject.toml

@@ -14,6 +14,7 @@ dependencies = [
     "datashader>=0.18.1",
     "defusedxml>=0.7.1",
     "gitpython>=3.1.44",
+    "gsw>=3.6.20",
     "hvplot>=0.11.3",
     "ipympl>=0.9.7",
     "jupyter>=1.1.1",