Improve modelbuilder notebook

[veenstrajelmer]

Short term (EDITO/Malta):

• auto download monthly phyc (include dataset_id in copernicus_get_datasetid())
• prevent daily files for monthly dataset
• add phyc, nitrate and phosphate, chlorofyl
• update waq settings to enable waq processes
• waq: HYDROLIB-core: support iniwaq* in ext file? >> ini files for waq parameters (include extrapolation?)
• add solarradiation/heatflux
• EDITO: Improve Delft3dfm modelbuilder workflow Deltares-research/EditoServices#1

Discuss waq processes:

• bloom requires different solarradiation aggregation? (LS/LV)
• Support D-Water Quality input in ExtOldModel HYDROLIB-core#551
• waq: write timfiles for source/sinks (for river discharges in large scale models and WAQ sources), e.g. p:\11209231-003-bes-modellering\hydrodynamica-waq\preprocessing\hydrology_runoff\hydrology_nutrient_to_tim_waq.py (by YC) >> bc files will be implemented in delft3dfm. sourcesinks are also used for river discharges.
• waq: HYDROLIB-core: Reader of substance file, including comparison to timfile columns. Example model in p:\11209251-sito-veerse-meer\model\runs_2019-2020\Veerse_meer_Model_to_subversion\dflowfm3d_dwaq-veerse_meer-j19_6-v1a_LH. Related issue: If old external forcing file references a pli file, the accompanying .tim/.cmp should also be read and stored HYDROLIB-core#501 (also contains column order of timfiles, so not super important if sourcesinks are forced via bc)
• waq: HYDROLIB-core: ignore or support waq statistics file (is defined in the substance file). statistics file often makes the model crash, so maybe find alternative.
• also wastewater: p:\11209231-003-bes-modellering\hydrodynamica-waq\preprocessing\wastewater_20231129\scripts\ (by LB) (is also sourcesinks)
• more WAQ things necessary: contact MW, FD, AM, LvdH, LV, LS

Discuss general:

• easier interaction with hydrolib-core (Sonia): "make keywords in ext/mdu files easier to access with dfm_tools. Flexibility"
• nans in ini sal/tem are filled per 1d dim (first lat, then lon), consider 2d interpolation/extrapolation: Extrapolation of CMEMS initial field #1021 (and consider filling source netcdfs for bc files first over lat/lon, instead of only filling bc over depth)
• if a polyline does not result in any plipoints, consider not writing it in extfile (FM raises an error in case of an ext entry for which the polygon does not have any forcing/bc blocks associated, happened in DSO project)
• make conversion_dict a global setting, so a custom conversion dict is automatically used everywhere in the modelbuilding process
• add GTSMip C3S obspoint locations #1121
• add ssc stations to map instead of arbitrary obspoints (coverage is not super good, but at least it makes sense) >> or more like add beaches, places, ports as observation points #748
• snap observation points to nearest cell #898
• adding river discharges (preferrably from a global public datasource) as source/sinks: GRDC has no API and no coastal points. Swedish SMHI EHYPE dataset?
• Fixes related to hydrolib-core issues #214

Further improvements:

• now using tpxo which requires large buffer size for plibnd, consider using EOT20 online dataset (not implemented) or tidemodel='GTSMv4.1_opendap' >> buffer can be reduced anyway since we also use nearest interp now
• prevent opendap gebco cutout link in modelbuilder notebook, no API is available, maybe via gebco ddb opendap. Or maybe add coarse dataset to opendap?
• prevent modelbuilder import, expose via dfm_tools directly
• support for waq in modelbuilder example script #504
• Minor modelbuilder improvements #602
• Fill nans in ini sal/tem netcdf #616
• Add initial fields for tracers #618
• CMEMS phyc not available in reanalysis #622
• Improve robustness of CMEMS/CDS authentication #603
• dsd improvements modelbuilder notebook #631
• modelbuilder print modeltree from dimr instead of mdu #725
• simplify cdsapi.Client() call after new cdsapi release #822
• Update to copernicus-marine-client #630
• Improvements in copernicus-marine-client related code #694
• Increase minimal copernicusmarine version #692
• Make conversion_dict an input parameter for the function cmems_nc_to_bc #838
• Add example with ETOPO online dataset #836
• remove os.listdir command after Show_tree output is not printed for old ext files HYDROLIB-core#581 is fixed
• connect to myint cmems reanalyis dataset also #861
• cmems_nc_to_ini produces netcdf file incoherent depth variable #944
• fix paths in dfmt.create_model_exec_files batfile #1029
• unique tide/cmems bc files with multiple polyfiles in modelbuilder #1070
• mdu.geometry.layertype in modelbuilder example script should be 2, also mdu.physics.temperature was defined twice
• prevent print of 2023-11-03 14:24:35,649 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/dummy >> quiet argument for Client class? >> not used anymore: unable to download ERA5 data using dfmt.download_ERA5 #971
• add 3D to notebook and discontinue py example script (also transfer othogonality visualization somewhere, linked p-drive in that script is now archived anyway): 1068 discontinue modelbuilder example python script #1069
• Disruptions and update to new cdsapi #739
• add function to overwrite several mdu keywords with coastal 2D/3D values #951
• prevent pydap connection if cmems files are already available, is difficult since we check for available times in products. This is not relevant anymore when we use copernicus-marine-client
• conversion of single plifile to multiplifile: Avoid duplicate polyline names in geodataframe_to_PolyFile #492
• interpolate_tide_to_bc paths for linux #1080 and pdrive windows should include slash #1084
• Support for noon and monthly averaged data in dfmt.cmems_nc_to_ini() #1086
• Consider 12-hour offset for CMEMS data #878
• notebook was improved in Expand modelbuilder notebook #463
• test model in windows exe and DeltaShell GUI, not all will work (e.g. lowercase keywords in bc-file are not properly read): Incompatible with Precompiled DFM #710
• contextily is still instable, maybe remove it from notebook
• Flexible varkeys in dfmt.preprocess_merge_meteofiles_era5() #887
• Consider time correction for ERA5 rainfall and more #899
• reduce ERA5 logging in cds_credentials() by using c=cdsapi.Client(quiet=True) after fix of Logging persists when 'quiet' parameter is reverted from True to False in Client class ecmwf/cdsapi#82
• CDS beta raises unclear error about accepting licenses ecmwf/cdsapi#108 (also registered under CUS-25567)
• Clearer errors in case of incorrect authentication ecmwf/cdsapi#137
• prevent red logging: https://stackoverflow.com/questions/25360714/avoid-red-background-color-for-logging-output-in-ipython-notebook
• consider using logging module for dfm_tools also, then reconfigure logging to include toolbox where it comes from
• add fes2022 tidal source #924
• disable many outputs via mdu settings, and consider using compression and single precision for his/mapfiles (but then the modelbuilder only works for new FM versions).
• suppress CMEMS and CDS logging in notebook (not in dfmt functions) with import logging; logging.getLogger("copernicus_marine_root_logger").setLevel(level="CRITICAL"). At least for getting time extents
• also support initial values in polygon?
• Add function to modelbuilder to generate observations along coastline #575
• add dry points (upon grid generation if still necessary): Non-orthogonal grid and illegal cells function #909
• add structures, cross-sections, thindams etc (quite complex to to this in a generic way, bit still add support for adding manually)
• set local source for tidal forcing #872
• DAP failure with GTSMv4.1_opendap tide source #613
• support for other crs/epsg than WGS84/4326. Requires reprojection of ERA5 data: https://corteva.github.io/rioxarray/stable/examples/reproject.html

Add sources for e.g. tide models, CMEMS and ERA5:

• UHSLC (public): https://uhslc.soest.hawaii.edu/
• IOC: https://www.ioc-sealevelmonitoring.org/
• GESLA3: https://gesla787883612.wordpress.com/
• FES2014 (not public, commercial paid): https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes/description-fes2014.html
• EOT20 (public): https://essd.copernicus.org/articles/13/3869/2021/
• GTSMv4.1 (public): https://www.deltares.nl/en/expertise/projects/global-modelling-of-tides-and-storm-surges

Add obspoints (or along coastline with #575):

obsfile = os.path.join(dir_output, f"{model_name}_obs.xyn")
obs_x = [-68.2240067, -68.2283020, -68.3852158, -68.1552582]
obs_y = [12.5409269, 12.2554617, 12.0076866, 12.0341187]
obs_n = ['Bonaire01', 'Bonaire02', 'Bonaire03', 'Bonaire04']
xynpoints = [hcdfm.XYNPoint(x=x,y=y,n=n) for x,y,n in zip(obs_x, obs_y, obs_n)]
xynmodel = hcdfm.XYNModel(points=xynpoints)
xynmodel.save(obsfile)

Running the model:

• generate sh file for snellius/h7/amazon, singularity? (bat for windows is handled, as well as generic dimr_config.xml)
• required: singularity, jupyter, maybe file browsing/editing (eg mobaxterm)
• download FM singularity via container harbour instead of download.py (OvD), currently from download portal: https://github.com/Deltares/dfm_tools/pull/488/files
• also make it possible to run the model without modelbuilder, so download+unpack zip with modelfiles?
• binder runs on Kubernetes
• C-scale used docker: https://github.com/c-scale-community/workflow-coastal-hydrowaq/tree/main/fm_model
• singularity on: https://download.deltares.nl/en/delft3dfm-2d3d-beta-singularity (direct download link available)
• add singularity container download #487
• singularity nor docker available on binder per default
• improve run_docker.sh #729
• Alternative: singularity also runs on AWS (RdG)
• Alternative: snellius supports interactive jupyter notebook, this should also work on h7 if a node can be reserved for the course
• Alternative: C-SCALE uses https://replay.notebooks.egi.eu/, replay is EGI's implementation of mybinder
• alternatively, EDITO uses jupyter and has connection to HPC
• also possible to use binderhub on different cluster
• if all else fails: zip+download model and run on local machine with bat file

Interactive jupyter slurm:

#!/bin/bash
#SBATCH -t 1:00:00
#SBATCH -N 1

# Make sure the jupyter command is available, either by loading the appropriate modules, sourcing your own virtual environment, etc.
module load 2021
source /home/muiss/miniconda3/etc/profile.d/conda.sh
conda activate /home/muiss/miniconda3/envs/era5-e

# Choose random port and print instructions to connect
PORT=`shuf -i 5000-5999 -n 1`
LOGIN_HOST=${SLURM_SUBMIT_HOST}-pub.snellius.surf.nl
BATCH_HOST=$(hostname)

echo "To connect to the notebook type the following command from your local terminal:"
echo "ssh -J ${USER}@${LOGIN_HOST} ${USER}@${BATCH_HOST} -L ${PORT}:localhost:${PORT}"
echo
echo "After connection is established in your local browser go to the address:"
echo "http://localhost:${PORT}"

jupyter lab --no-browser --port ${PORT}