Masking/Cropping Meteo data makes data_handling/writenetcdf not work correctly

[dubeysarth]

I was exploring how to set the model up for regions of interest using the observation data for the variables available and climate data for those that aren't. I keep encountering this error:
AttributeError: _FillValue attribute must be set when variable is created (using fill_value keyword to createVariable)
I felt it might be a metadata issue.
So, I then used the data for GSWP3-W5E5 at 30arcmins with (StepStart, SpinUp, StepEnd) = (01/01/2001, 01/01/2002, 31/12/2005). I executed the following 5 runs for Krishna basin:
(a) All variables as-is.
(b) 'pr' cropped for India [lon = slice(66,100), lat = slice(39, 6)], so no lat/lon coordinates outside this box are in NC file
(c) 'pr' masked for India with NaN outside box
(d) 'pr' masked for India with 0 outside box
(e) 'pr' values within India replaced with interpolation of observed at that grid (checked that it wasn't NaNs)
Under the same .ini file, the (a) works properly, while (b-e) raises the same error as above.

I have been able to use the model so far, achieve calibration and validation in Danube, and get some outputs for Indian basins using GSWP3-W5E5.
I have 2 queries:

1. Are you able to reproduce this error?
2. I was assuming it only extracts the meteo and static params for cells that are identified using MaskMap and LDD, is that wrong? If not, why does it not operate the same for all 5 runs (a-e)?
3. Does making this work require modifying anything in CWatM-Earth-30mins? Or am I missing something obvious?

The exact log is as follows:
File "CWatM/CWatM/run_cwatm.py", line 39, in <module> main(settings, args) File "CWatM/CWatM/cwatm/run_cwatm.py", line 334, in main success, last_dis = CWATMexe(settingsfile[0]) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "CWatM/CWatM/cwatm/run_cwatm.py", line 149, in CWATMexe stCWATM.run() File "CWatM/CWatM/cwatm/management_modules/dynamicModel.py", line 45, in run self.step() File "CWatM/CWatM/cwatm/management_modules/dynamicModel.py", line 31, in step self._model.dynamic() File "CWatM/CWatM/cwatm/cwatm_dynamic.py", line 154, in dynamic self.output_module.dynamic() File "CWatM/CWatM/cwatm/management_modules/output.py", line 436, in dynamic outMap[map][i][2] = writenetcdf(netfile, varname,"", "undefined", eval(inputmap), dateVar['currDate'],dateVar['currwrite'], flag, True, dateVar['diffdays']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "CWatM/CWatM/cwatm/management_modules/data_handling.py", line 1405, in writenetcdf exec('%s="%s"' % ("longitude." + i, metadataNCDF['lon'][i])) File "<string>", line 1, in <module> File "src/netCDF4/_netCDF4.pyx", line 4878, in netCDF4._netCDF4.Variable.__setattr__ AttributeError: _FillValue attribute must be set when variable is created (using fill_value keyword to createVariable)

The first file it tries to create for me is 'Rain_daily.nc' and it shows weird lon values which are
array([9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36, 9.96920997e+36])

PS: I tried making edits to data_handling by using set_fill_off, adding fill_value to each createVariable but the error persisted.

[dubeysarth]

I was able to figure out the issue. Anyone who bumps into something similar, kindly note that you would have to use netCDF instead of xarray to write the .nc files. When xarray's to_netcdf is used, the metadata changes. An example is below,

• ds.variables for GSWP3-W5E5 returns
  {'lon': <class 'netCDF4._netCDF4.Variable'> float64 lon(lon) standard_name: longitude long_name: Longitude units: degrees_east axis: X unlimited dimensions: current shape = (720,) filling on, default _FillValue of 9.969209968386869e+36 used, 'lat': <class 'netCDF4._netCDF4.Variable'> float64 lat(lat) standard_name: latitude long_name: Latitude units: degrees_north axis: Y unlimited dimensions: current shape = (360,) filling on, default _FillValue of 9.969209968386869e+36 used, 'time': <class 'netCDF4._netCDF4.Variable'> float64 time(time) standard_name: time long_name: Time units: days since 1900-1-1 00:00:00 axis: T calendar: proleptic_gregorian unlimited dimensions: time current shape = (3652,) filling on, default _FillValue of 9.969209968386869e+36 used, 'pr': <class 'netCDF4._netCDF4.Variable'> float32 pr(time, lat, lon) standard_name: precipitation_flux long_name: Precipitation units: kg m-2 s-1 _FillValue: 1e+20 missing_value: 1e+20 unlimited dimensions: time current shape = (3652, 360, 720) filling on}
• ds.variables after any modification saved using xarray's to_netcdf() returns
  {'pr': <class 'netCDF4._netCDF4.Variable'> float32 pr(time, lat, lon) _FillValue: 1e+20 standard_name: precipitation_flux long_name: Precipitation units: kg m-2 s-1 missing_value: 1e+20 unlimited dimensions: current shape = (3652, 360, 720) filling on, 'lon': <class 'netCDF4._netCDF4.Variable'> float64 lon(lon) _FillValue: nan standard_name: longitude long_name: Longitude units: degrees_east axis: X unlimited dimensions: current shape = (720,) filling on, 'lat': <class 'netCDF4._netCDF4.Variable'> float64 lat(lat) _FillValue: nan standard_name: latitude long_name: Latitude units: degrees_north axis: Y unlimited dimensions: current shape = (360,) filling on, 'time': <class 'netCDF4._netCDF4.Variable'> float64 time(time) _FillValue: nan standard_name: time long_name: Time units: days since 1900-1-1 00:00:00 axis: T calendar: proleptic_gregorian unlimited dimensions: current shape = (3652,) filling on}