Total water storage reporting

This release adds the computation and reporting of total water storage maps.

Replacement of cython with numba (routing)

• use of numba instead of cython for parallel computations in the routing module: numba was already used in other parallelized functions in soil modules. Now this version extends the use of numba to the kinematic wave parallel routing module. The change allows to completely exclude the use of Cython in the whole Lisflood OS software with no performance loss and skipping further compilation steps and potential run issues when porting kinematic wave compiled library to other systems/hardware. Numba use a Just in time (JIT) compilation that compiles python routines at runtime, generating optimized code and allowing parallelization by skipping the python Global Interpreter Lock (GIL) of specific parallelizable computation functions.

• xml files and documentation files are now updated removing the variables used by kinematic wave parallel routing module and skipping the steps for the compilation of Cython routing functions. Numba parameters in xml files should be used instead of the old kinematic wave ones to allow or disable parallelization on multiple CPUs hardware systems.

Improved NetCDF4/PCRaster outputs module

• Outputs module refactored to be more efficient and modular.
• New option "OutputMapsChunks" added, allowing to write steps in chunks in the NetCDF4 file. The option "OutputMapsChunks" requires as input a positive integer number X, meaning that the code will dump outputs to disk every X steps (default value = 1).
• New option "OutputMapsDataType" added, allowing to choose the output maps data type between float64 and float32 (can be used to save space in output maps).
• Added support to flipped input static maps and meteo maps
• The Lisflood OS version number is now recorded in the metadata of the NetCDF4 files. Institution and author metadata are updated according to the version of Lisflood used to generate the output files.
• Flags to check the maps: now lisflood command can be executed to check the correctness of input maps before running the model. The option -c is available as an additional command line argument (e.g. "lisflood settings.xml -c") and generates the table of all the input maps specified in the input xml file. The table will show name of the map, the file path (or single value of the variable), the number of non missing values (nonMV) and missing values (MV) after the comparison with the mask map and LDD map, minimum, maximum and average values for each map. In case any map have missing values, a warning will be issued on screen.
• The "ReportSteps" option now support formulas. The "ReportSteps" option available in XML file can be used to customize the output frequency of the model state variables. The desired time steps can be specified by using ranges, formulas and parameters as indicated in the documentation here: https://ec-jrc.github.io/lisflood-code/3_step3_preparing-setting-file. Note that this option only impacts the output frequency of the model state variables (activated by the "repStateMaps" option), not to the auxiliary variables. The full list of the affected variables is here: https://ec-jrc.github.io/lisflood-code/4_annex_state-variables/
• Lisflood OS User Guide documentation is now included into the same repository of the Lisflood software into the "docs" folder: in this way the documentation will be up to date with the corresponding lisflood release and users can access to it offline. The documentation files are written in Markdown (MD) format and are visible using a Markdown viewer locally (available on most IDE like Visual Studio, Notepad++, etc...). The documentation can be also rendered locally as a localhost website using Jekyll and Ruby.

Valid_min and valid_max value outilers filtering in meteo input maps

This release includes the following fixes:

• XArray was not filtering out values outside valid_min and valid_max range in meteo maps. Now outliers values, if any, will be replaced by NaNs.
• added flag "-s" to ignore values out of valid range in meteo maps.
• fixed issue related to variables order in reading netcdf files
• fixed pandas parse_time_string renamed function issue.
• corrected units in netcdf metadata of Theta output maps

Coords from netcdf template

Small fix to use the netcdf template to set the coordinates in the netcdf outputs rather than using a pcraster clone.
This avoids inconsistencies between the netcdf inputs and outputs.

Hot fix to increase the robustness of the code

This release features the following changes:

• a hot fix to prevent the warm start (i.e. restarting from a previous state) to occasionally fail due to small approximation errors (order of magnitude: 10exp-12) when writing and reading the state files;
• updates to the reporting options for output maps. List of repStateMaps files is now the same as repEndMaps and it only contains files used to warm start the model. Additional reporting options are available to singularly output maps of different model variables.

Bug fix: corrected updating of the reservoirs and lakes volume after water abstraction for anthropogenic use

This release features the following changes:

1. The volume of lakes and reservoirs is correctly updated when water is abstracted for anthropogenic use.

2. The initialization run (or prerun; activated by switching on the "InitLisflood" option) implements a consistent solution throughout the code for the water abstraction module. As in v.4.0.1, the warm start of the initialization run is fully functional.

3. A hot fix to prevent the warm start (i.e. restarting from a previous state) to occasionally fail due to small approximation errors (order of magnitude: 10exp-12) when writing and reading the state files.

The changes (1) and (2) of this release have impacts on the results of LISFLOOD-OS when the following three optional modules are simultaneously activated: reservoirs, lakes, water use.
The magnitude of the impact of the above changes mainly depends on the magnitude of the water demand for energy use, and on the extent of the rice fields within the modelled area.

Waterbalance unit test and warm start in initialization run

• The changes of this release allow the warm start of the initialization run (or prerun; <setoption choice="1" name="InitLisflood"/>).
  More specifically, the changes of this release allow to print the end step of all the state files required to correctly perform the warm start of the initilization run.
  Before this release, the warm start could be correctly performed only for the run (<setoption choice="0" name="InitLisflood"/>).

• Furthermore, a unit test for waterbalance has been added.

Revised variables structure and improved pixel-wise computations for the soil loop

LISFLOOD-OS v4.0.0 includes the following changes:

1. pixel-by-pixel computation of water infiltration into the soil within the soilloop.py module.
2. changes in the allocation of water abstraction.
3. patches in the modules riceirrigation.py and frost.py to avoid non-realistic results in specific conditions.
4. improvements in the computation of the mass balance.
5. bug fixes in the use of the transientlandusechange option, in the use of the option simulatePF, in the writing of the monthly outputs.

Moreover, to ensure compatibility with LISFLOOD-EPIC, modules source code is compatible with XArray variables, and XArray variables will be allocated when EPIC module is ON.
LISFLOOD-EPIC has been developed by JRC-D2. When compared to LISFLOOD v3.2.0, LISFLOOD-EPIC enables a more detailed representation of crop growth and irrigation. References: ASR - Assessing groundwater irrigation sustainability in the Euro-Mediterranean region with an integrated agro-hydrologic model (copernicus.org) , Gelati, E. et al.: Integrated model of irrigation, crop growth and water resources (LISFLOOD-EPIC): description and application in the Euro-Mediterranean area, in preparation, 2020.
Important note: EPIC modules are not yet available from the OS-LISFLOOD repository.

1. computation of water infiltration into the soil within the soilloop.py module.
   LISFLOOD v3.2.0: all the pixels of a land use fraction are allocated to a vector and the number of loops is defined by the most critical pixel. More specifically, all the pixels within the computational domain are allocated the number of iterations that are necessary to allow the numerical stability of the most critical pixel. The number of iterations consequently depends on the specific computational domain. This can lead to different results when modelling sub-catchmennts and entire basins with highly heterogenous soils.
   LISFLOOD v4.0.0: pixel by pixel computation and parallelization of the computations by using the python package numba.
   Benefits of v4.0.0: each pixel is allocated the number of iterations that allow its numerical stability, this guarantees the consistency of results when modelling sub-catchments and the entire basin with highly heterogeneous soils. Decreased computational time due to parallel computations.

2. changes in the allocation of water abstraction.
   LISFLOOD v3.2.0: direct abstraction of the consumptive water use.
   LISFLOOD v4.0.0: abstraction of the demanded water volume for each use (industrial, domestic, energy, livestock); the consumptive water use leaves the system; the unused water volume is returned to the channels.
   Minor note: small differences in the computation of the leakages.

3. patches in the modules riceirrigation.py and frost.py to avoid non-realistic results in specific conditions.
   riceirrigation.py: LISFLOOD v3.2.0: in presence of a very thick third soil layer (10^2 m) the drainage of the rice fields causes non realistic large flow discharge values in the channels.
   LISFLOOD v4.0.0: only soil layers 1a and 1b are used by the rice module (for both the saturation and the drainage phases). By definition, the second soil layer includes the roots depth.
   frost.py: LISFLOOD v3.2.0: there is no upper boundary to the frost index value. Rain events on deeply frozen soils are totally converted into runoff and can cause false alarms.
   LISFLOOD v4.0.0: the maximum frost index value is set to 57.

4. improvements in the computation of the mass balance.
   LISFLOOD v3.2.0: the mass balance is computed by considering the input and output volumes up to the current computational step. Small numerical errors accumulate over time and give the erroneous impression of large mass balance errors.
   LISFLOOD v4.0.0: computation of the mass balance by considering the single computational step.

5. bug fixes in the use of the transientlandusechange option, in the use of the option simulatePF, in the writing of the monthly outputs.
   Correct use of the rice fraction by landusechange.py DYNAMIC; use of the correct soil water content variable for the computation of pF1 (ws1a replaced ws1); correct reporting of the monthly time steps by the optional module indicatorcalc.py.

Lisflood 4.0.0 is the version used for EFAS 5 and GloFAS 4 calibration.

New XArray reader

This release includes the following changes:

• a new xarray reader for the water abstraction demand maps;
• a unit test to verify the functioning of the reader for the water abstraction maps;
• bug fixes and improvements to the caching function and the chunking function;
• improvements of the management of latitude and longitude grids to allow maximum precision of the reference system;
• improvements of the definition of the output variables.

All of the changes above improve the code capability to handle input and outputs.
The modelling of the hydrological processes has not been changed.

IMPORTANT NOTE: the results of the unit tests of this release are different from the results of the unit tests of the release 3.1.1. The differences are due exclusively to a different use of of the optional moduels (e.g. groundwatersmooth) within the .xml setttings file of the unit tests. When using the same settings of .xml file, and the same dataset, v3.1.1 and v3.2.0 provide the same results.