Update get_onc_ctd and get_onc_ferry workers to work with ONC API…

… v3 (#234)

* Modernize test_get_onc_ctd

Replace unittest.mock.patch decorator with pytest.fixture for mock worker.

Add unit tests for production YAML config file elements related to worker;
re: issue #117.

Replace unittest.mock.patch decorator with pytest caplog fixture for tests of
logging; re: issue #82.

* Add unit test for YAML config file elements

Added unit test for production YAML configuration file ctd data observations
elements used in after_* () functions.

* Improve get_onc_ctd.main() function

Update docstring re: issue #121.

Return worker so that modernized unit tests for main() work; re issue #81.

* Update get_onc_ctd worker to use ONC API v3

Modified the get_onc_ctd worker to use ONC API v3 'getByLocation' method. The
new method uses a location code instead of a station name. Also, these changes
include adding a 'dateTo' parameter to limit the data to a specific time range
for the day. Additionally, small changes to variable and parameter names were
made to match the new method requirements.

* Remove test skipping due to resolved issue #174

The pytest skip marker for "_resample_nav_coord()" function in
"tests/workers/test_get_onc_ferry.py" file has been removed. This has happened
following the successful resolution of issue number #174, making it unnecessary
to skip these specific unit tests anymore.

Fixes issue #174

* Add unit tests for YAML config file elements

Add unit tests for production YAML config file elements related to worker;
re: issue #117.

* Update YAML config with Tsawwassen - Duke Point ferry route

The 'nowcast.yaml' configuration file has been updated to include the
Tsawwassen - Duke Point ferry route details including devices, sensors,
and other relevant data.

* Update dataset ID for TWDP-ferry

The dataset ID for TWDP-ferry was updated in the 'nowcast.yaml' configuration
file and in respective test. The change is reflected accurately to match the
new details for the Tsawwassen - Duke Point ferry route.

* Update get_onc_ferry worker to use ONC API v3

Modified the get_onc_ferry worker to use ONC API v3 'getByLocation' method. The
new method uses a location code instead of a station name. Also, these changes
include adding a 'dateTo' parameter to limit the data to a specific time range
for the day. Server-side averaging into 1-second bins is used to ensure that an
entire day's observations from each sensor can be obtained by a single API
request; there is a limit of 100,000 "rows" per sensor. Additionally, small
changes to variable and parameter names were made to match the new method
requirements.

* Update ONC_data_product_url attr in ferry datasets

Updated the ONC_data_product_url attribute in the datasets produced by the
get_onc_ferry worker to reflect changes in the data source's domain name and API
query parameter names. Code changes include updating the ONC data API domain
name, adding the 'locationCode' query parameter, and changing the
'deviceCategory' query parameter name to 'deviceCategoryCode'. All of those
changes are for compatibility with the ONC data API v3.

* Correct representation of relative humidity attribute

Changed the naming representation of the relative humidity attribute in the
'get_onc_ferry' worker. The attribute was initially named "REL_HUMIDITY" and it
was renamed to "rel_humidity", matching the common low-caps format of the other
attributes for consistency of naming convention.

* Update QA/QC filter criteria in get_onc_ferry worker

Adjust the QA/QC filter settings in the 'get_onc_ferry' worker to now include
data where the 'qaqcFlag' attribute is less than or equal to 1 or greater than
or equal to 7. This update is necessary because of the change to server-side
averaging in the request.

* Improve empty data array handling

ONC API v3 has more/different ways of returning empty responses to data requests
that we have to handle. It's not as clean as I would like, but it is good
enough for the purpose.

config/nowcast.yaml

@@ -245,7 +245,7 @@ temperature salinity:
 
 
 observations:
-  # Dataset that maps high resolution lon/lat grid on to NEMO j/i indices
+  # Dataset that maps high-resolution lon/lat grid on to NEMO j/i indices
   lon/lat to NEMO ji map: /SalishSeaCast/grid/grid_from_lat_lon_mask999.nc
   # ONC Strait of Georgia nodes real-time CTD data
   ctd data:
@@ -259,12 +259,69 @@ observations:
     dest dir: /results/observations/ONC/CTD/
     # Template for ONC CTD T&S data file path
     # **Must be quoted to project {} characters**
-    filepath template: '{station}/{station}_CTD_15m_{yyyymmdd}.nc'
+    filepath template: "{station}/{station}_CTD_15m_{yyyymmdd}.nc"
 
   # ONC Strait of Georgia ferry platforms real-time data
   ferry data:
-    ferries: {}
-      # see PR#109 for TWDP ferry example of this section and corresponding config tests
+    ferries:
+      # Tsawwassen - Duke Point route
+      TWDP:
+        route name: Tsawwassen - Duke Point
+        ONC station description: Mobile Platforms, British Columbia Ferries, Tsawwassen - Duke Point
+        location:
+          # ONC scalardata.getByStation API query parameters:
+          #
+          # ONC station identifiers to use for nav data, in priority order
+          stations:
+            - TWDP.N1
+            - TWDP.N2
+          device category: NAV
+          sensors:
+            - longitude
+            - latitude
+          terminals:
+            # Terminal names from salishsea_tools.places.PLACES
+            - Tsawwassen
+            - Duke Pt.
+        devices:
+          # ONC device category
+          TSG:
+            # device sensor names
+            sensors:
+              # ERDDAP sensor name: ONC sensor name
+              temperature: temperature
+              conductivity: conductivity
+              salinity: salinity
+          OXYSENSOR:
+            sensors:
+              o2_saturation: oxygen_saturation
+              o2_concentration_corrected: oxygen_corrected
+              o2_temperature: temperature
+          TURBCHLFL:
+            sensors:
+              cdom_fluorescence: cdom_fluorescence
+              chlorophyll: chlorophyll
+              turbidity: turbidity
+          CO2SENSOR:
+            sensors:
+              co2_partial_pressure: partial_pressure
+              co2_concentration_linearized: co2
+          TEMPHUMID:
+            sensors:
+              air_temperature: air_temperature
+              relative_humidity: rel_humidity
+          BARPRESS:
+            sensors:
+              barometric_pressure: barometric_pressure
+          PYRANOMETER:
+            sensors:
+              solar_radiation: solar_radiation
+          PYRGEOMETER:
+            sensors:
+              longwave_radiation: downward_radiation
+        # Template for ONC ferry data file path
+        # **Must be quoted to project {} characters**
+        filepath template: "{ferry_platform}/{ferry_platform}_TSG_O2_TURBCHLFL_CO2_METEO_1m_{yyyymmdd}.nc"
     # Destination directory for ONC ferry data netCDF files
     dest dir: /results/observations/ONC/ferries/
 
@@ -274,7 +331,7 @@ observations:
     csv dir: /opp/observations/AISDATA/
     # Destination directory for VFPA HADCP data netCDF files
     dest dir: /opp/observations/AISDATA/netcdf/
-    # Template for VFPA HADCP data file path
+    # Template for the VFPA HADCP data file path
     # **Must be quoted to project {} characters**
     filepath template: 'VFPA_2ND_NARROWS_HADCP_2s_{yyyymm}.nc'
 
@@ -429,7 +486,7 @@ erddap:
 #      out of service since 22-Dec-2019; repair ETA unknown
 #      - ubcONCUSDDLCTD15mV1
     TWDP-ferry:
-      - ubcONCTWDP1mV1
+      - ubcONCTWDP1mV18-01
     nowcast-green:
       - ubcSSg3DBiologyFields1hV21-11
       - ubcSSg3DLightFields1hV21-11

nowcast/workers/get_onc_ctd.py

@@ -42,9 +42,7 @@
 
 
 def main():
-    """Set up and run the worker.
-
-    For command-line usage see:
+    """For command-line usage see:
 
     :command:`python -m nowcast.workers.get_onc_ctd -h`
     """
@@ -61,6 +59,7 @@ def main():
         help="UTC date to get ONC node CTD data for.",
     )
     worker.run(get_onc_ctd, success, failure)
+    return worker
 
 
 def success(parsed_args):
@@ -80,15 +79,15 @@ def failure(parsed_args):
 def get_onc_ctd(parsed_args, config, *args):
     ymd = parsed_args.data_date.format("YYYY-MM-DD")
     logger.info(f"requesting ONC {parsed_args.onc_station} CTD T&S data for {ymd}")
-    TOKEN = os.environ["ONC_USER_TOKEN"]
     onc_data = data_tools.get_onc_data(
         "scalardata",
-        "getByStation",
-        TOKEN,
-        station=parsed_args.onc_station,
-        deviceCategory="CTD",
-        sensors="salinity,temperature",
+        "getByLocation",
+        os.environ["ONC_USER_TOKEN"],
+        locationCode=parsed_args.onc_station,
+        deviceCategoryCode="CTD",
+        sensorCategoryCodes="salinity,temperature",
         dateFrom=data_tools.onc_datetime(f"{ymd} 00:00", "utc"),
+        dateTo=data_tools.onc_datetime(f"{ymd} 23:59", "utc"),
     )
     try:
         ctd_data = data_tools.onc_json_to_dataset(onc_data)