diff --git a/docs/configuration.rst b/docs/configuration.rst index 6462524..83192bb 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -4,7 +4,7 @@ Configuration Specifications ============================ This page documents the various options and parameters that can be set in the configuration -file. +file. An example configuration file can be found on the examples page. MetSim Section -------------- @@ -108,23 +108,47 @@ available at: ``. For more information about the "triangle" method see :doc:`PtriangleMethod.pdf`. For more information about input and output variables see the :ref:`data` page. +:: + + # Comments begin with hashtags + # The first non-comment line must begin with the following: + Metsim: + time_step: int + start: YYYY-MM-DD + stop: YYYY-MM-DD + + # Paths to input files + forcing: str + domain: str + state: str + + # Output file specification + out_dir: str + out_prefix: str + + # Algorithmic controls + utc_offset: bool + prec_type: str + lw_type: str + lw_cloud: str + chunks section -------------- The ``chunks`` section describes how parallel computation should be grouped in space. For example, to parallelize over 10 by 10 chunks of latitude and longitude (with netcdf dimensions named ``lat`` and ``lon``, respectively) you would use: +:: -.. code-block:: ini - [chunks] - lat = 10 - lon = 10 + chunks: + lat: 10 + lon: 10 Alternatively, for an HRU based run chunked into 50 element jobs you would use: +:: -.. code-block:: ini - [chunks] - hru = 50 + chunks: + hru: 50 As a general rule of thumb, try to evenly chunk the domain in such a way that the number of jobs to run is some multiple of the number of processors you wish @@ -132,45 +156,15 @@ to run on. forcing_vars and state_vars section --------------- -The ``forcing_vars`` and ``state_vars`` sections are where you can specify which variables are in your -input data, and the corresponding symbols which MetSim will recognize. The -format of this section depends on the value given in the ``in_fmt`` entry in -the ``MetSim`` section of the configuration file. See below for conventions for -each input type. - - -netcdf and data -``````````````` -The ``in_vars`` section for NetCDF and xarray input acts as a mapping between the variable +The ``forcing_vars`` and ``state_vars`` sections are where you can specify which +variables are in your input data, and the corresponding symbols which MetSim will +recognize. The ``in_vars`` section for acts as a mapping between the variable names in the input dataset to the variable names expected by MetSim. The format -is given as ``metsim_varname = netcdf_varname``. The minimum required variables +is given as ``metsim_varname: netcdf_varname``. The minimum required variables given have ``metsim_varname``\s corresponding to ``t_min``, ``t_max``, and ``prec``; these variable names correspond to minimum daily temperature (Celcius), maximum daily temperature (Celcius), and precipitation (mm/day). -ascii -````` -The ``in_vars`` section for ASCII input acts similarly to the NetCDF input -format, except for one key point. Variables should be given as a tautology: the -format is given as ``metsim_varname = metsim_varname``. The order that the -variables are given corresponds to the column numbers that they appear in the -input files. The minimum required variables are ``t_min``, ``t_max``, and -``prec``; these variable names correspond to minimum daily temperature (Celcius), -maximum daily temperature (Celcius), and precipitation (mm/day). - -binary -`````` -This section has an input style for binary files that mimics the VIC version 4 -input style. Each line is specified as ``varname = scale cdatatype``, where -``varname`` is the name that MetSim should use for the column, ``scale`` is a -floating point scaling factor that should be applied after conversion from -binary to floating point; the conversion applied by the ``scale`` is applied -after the value in the input is converted from binary to the ``cdatatype`` -specified for each variable. Valid ``cdatatype``\s are ``signed`` and -``unsigned``. ``signed`` values are interpreted as values which can be positive -or negative, whereas ``unsigned`` values are interpreted as values that can only -be greater than or equal to zero. - domain_vars section ------------------- The ``domain_vars`` section is where information about the domain file is given. @@ -184,26 +178,109 @@ a mask of valid cells in the domain, and the elevation given in meters. If including ``dur`` and ``t_pk`` for disaggregating daily precipitation according to the "triangle" method. +out_vars section +---------------- +The ``out_vars`` section is where you can specify the output variables that you +want to include. There are two formats for this section. The first is the old format, +which we provide backwards compatibility for. You simply specify in the top level +``[MetSim]`` section a list of output variables with the names used by MetSim. They +will be written out with the same names used internally. Available options are +dependent on whether daily or subdaily output is being generated. Options for +daily output are: + +- pet +- shortwave +- t_max +- t_min +- tskc + +Options for subdaily output are: + + - prec + - shortwave + - longwave + - temp + - vapor_pressure + - air_pressure + - tskc + - rel_humid + - spec_humid + - wind + +The syntax for output specification is as follows: +:: + + out_vars: + metsim_varname: + out_name: str + units: str + +unit conversions +================ +The ``out_vars`` section allows for specification of some simple unit conversions +for MetSim output. The allowed options are as follows (invalid options will revert +to the default after issuing a warning): + + * prec + - ``mm timestep-1`` (default) + - ``mm s-1`` + - ``mm h-1`` + * pet (daily output only) + - ``mm timestep-1`` (default) + - ``mm s-1`` + - ``mm h-1`` + * t_max (daily output only) + - ``C`` (default) + - ``K`` + * t_min (daily output only) + - ``C`` (default) + - ``K`` + * temp + - ``C`` (default) + - ``K`` + * vapor_pressure + - ``kPa`` (default) + - ``hPa`` + - ``Pa`` + * air_pressure + - ``kPa`` (default) + - ``hPa`` + - ``Pa`` + * tskc (cloud fraction) + - ``fraction`` (default) + - ``%`` + * rel_humid + - ``%`` (default) + - ``fraction`` + constant_vars section ------------------- -The ``constant_vars`` section is optional and allows you to set some of the -forcing inputs to a constant value. The specification simply consists of entries -of the form ``metsim_varname = value``, where ``value`` is a number that can be -converted to a double. There can only be one entry per line. If the +The ``constant_vars`` section is optional and allows you to set some of the +forcing inputs to a constant value. The specification simply consists of entries +of the form ``metsim_varname: value``, where ``value`` is a number that can be +converted to a double. There can only be one entry per line. If the ``metsim_varname`` corresponds to an entry that is already in the ``forcing_vars`` -section, then the constant value will take precedence. In the current +section, then the constant value will take precedence. In the current implementation there must be at least one non-constant entry in ``forcings_vars`` (i.e. at least one entry that is not also in ``constant_vars``). For example: -``wind = 2.0`` +:: + + constant_vars: + wind: 2.0 + will result in a constant wind field in the output file. In this case ``wind`` does not need to be specified in the ``forcing_vars`` section. If it was, it will still be set to a constant value of 2 m/s. Similarly: -``t_max = 30 -t_min = 10`` +:: + + constant_vars: + t_max = 30.0 + t_min = 10.0 + will result in output with a diurnal cycle in which the temperature varies at -all locations between 10C and 30C. However, all estimation and disaggregation -routines are still evaluated, with constant ``t_max`` and ``t_min`` as input. \ No newline at end of file +all locations between 10C and 30C. However, all estimation and disaggregation +routines are still evaluated, with constant ``t_max`` and ``t_min`` as input. diff --git a/docs/configuration_ini.rst b/docs/configuration_ini.rst new file mode 100644 index 0000000..8c507b7 --- /dev/null +++ b/docs/configuration_ini.rst @@ -0,0 +1,252 @@ +.. _configuration_ini: + +INI Configuration Specifications +================================ + +.. warning:: This page provides documentation only for backwards compatibility. + New users should use the YAML configuration file format, as it is more flexible + and is where new features will be added. The INI format will be depreciated in + version 3 of MetSim. + +This page documents the various options and +parameters that can be set in the configuration +file. + +MetSim Section +-------------- + +**Required Variables** + +``time_step :: int``: The timestep to disaggregate in minutes. If given as 1440 +(number of minutes in a day), no disaggregation will occur. This value must +divide 1440 evenly. + +``start :: str``: The time to start simulation given in the format +``yyyy/mm/dd`` + +``stop :: str``: The time to end simulation given in the format +``yyyy/mm/dd``. + +``forcing :: path``: The path to the input forcing file(s). See the section +on __forcing_vars__ for more details. + +``domain :: path``: The path to the input domain file. See the section on +__domain_vars__ for more details. + +``state :: path``: The path to the input state file. + +``out_dir :: path``: The location to write output to. If this path doesn't +exist, it will be created. + +``forcing_fmt :: str``: A string representing the type of input files specified in +the ``forcing`` entry. Can be one of the following: ``ascii``, ``binary``, +``netcdf``, or ``data``. + +**Optional Variables** + +``out_prefix :: str``: The output file base name. Defaults to ``forcing``. + +``out_precision :: str``: Precision to use when writing output. Defaults to +``f8``. Can be either ``f4`` or ``f8``. + +``time_grouper :: str``: Whether to chunk up the timeseries into pieces for +processing. This option is useful to set for when you are limited on +memory. Each chunk of output is written as ``{out_prefix}_{date_range}`` when +active. Any valid ``pandas.TimeGrouper`` string may be used (e.g. use '10AS' +for 10 year chunks). + +``verbose :: bool``: Whether to print output to ``stdout``. Should be set using +the ``-v`` flag for command line usage. This can be set for scripting purposes, +if desired. Set to ``1`` to print output; defaults to ``0``. + +``sw_prec_thresh :: float``: Minimum precipitation threshold to take into +account when simulating incoming shortwave radiation. Defaults to ``0``. + +``rain_scalar :: float``: Scale factor for calculation of cloudy sky +transmittance. Defaults to ``0.75``, range should be between ``0`` and +``1``. + +``utc_offset :: bool``: Whether to use UTC timecode offsets for shifting +timeseries. Without this option all times should be considered local to +the gridcell being processed. Large domain runs probably want to set this +option to ``True``. + +``lw_cloud :: str``: Type of cloud correction to longwave radiation to apply. +Can be either ``DEFAULT`` or ``CLOUD_DEARDORFF``. Defaults to +``CLOUD_DEARDORFF``. Capitalization does not matter. + +``lw_type :: str``: Type of longwave radiation parameterization to apply. Can be +one of the following: ``DEFAULT``, ``TVA``, ``ANDERSON``, ``BRUTSAERT``, +``SATTERLUND``, ``IDSO``, or ``PRATA``. Defaults to ``PRATA``. Capitalization +does not matter. + +``tdew_tol :: float``: Convergence criteria for the iterative calculation of +dewpoint temperature in MtClim. Defaults to ``1e-6``. + +``tmax_daylength_fraction :: float`` : Weight for calculation of time of maximum +daily temperature. Must be between ``0`` and ``1``. Defaults to ``0.67``. + +``tday_coef :: float``: Scale factor for calculation of daily mean temperature. +Defaults to ``0.45``, range should be between ``0`` and ``1``. + +``lapse_rate :: float``: Used to calculate atmospheric pressure. Defaults to +``0.0065 K/m``. + +``out_vars :: list`` : List of variables to write to output. Should be a list +containing valid variables. The list of valid variables is dependent on which +simulation method is used, as well as whether disaggregation is used. Defaults +to ``['temp', 'prec', 'shortwave', 'longwave', 'vapor_pressure', 'red_humid']``. + +``prec_type :: str``: Type of precipitation disaggregation method to use. Can be +one of the following: ``uniform``, ``triangle``, or ``mix``. Defaults to +``uniform``. Capitalization does not matter. Under ``uniform`` method, +precipitation is disaggregated by dividing uniformly over all sub-daily +timesteps. Under ``triangle`` the "triangle" method is employed whereby daily +precipitation is distributed assuming an isosceles triangle shape with peak and +width determined from two domain variables, ``t_pk`` and ``dur``. Under +``mix``, the "uniform" method is used on days when ``t_min`` < 0 C, and +"triangle" is used on all other days; this hybrid method retains the improved +accuracy of "triangle" in terms of warm season runoff but avoids the biases +in snow accumulation that the "triangle" method sometimes yields due to fixed +event timing within the diurnal cycle of temperature. A domain file for the +CONUS+Mexico domain, containing the ``dur`` and ``t_pk`` parameters is +available at: ``. For more +information about the "triangle" method see :doc:`PtriangleMethod.pdf`. + +For more information about input and output variables see the :ref:`data` page. + +chunks section +-------------- +The ``chunks`` section describes how parallel computation should be grouped +in space. For example, to parallelize over 10 by 10 chunks of latitude and +longitude (with netcdf dimensions named ``lat`` and ``lon``, respectively) you would use: + +.. code-block:: ini + [chunks] + lat = 10 + lon = 10 + +Alternatively, for an HRU based run chunked into 50 element jobs you would use: + +.. code-block:: ini + [chunks] + hru = 50 + +As a general rule of thumb, try to evenly chunk the domain in such a way that +the number of jobs to run is some multiple of the number of processors you wish +to run on. + +forcing_vars and state_vars section +--------------- +The ``forcing_vars`` and ``state_vars`` sections are where you can specify which variables are in your +input data, and the corresponding symbols which MetSim will recognize. The +format of this section depends on the value given in the ``in_fmt`` entry in +the ``MetSim`` section of the configuration file. See below for conventions for +each input type. + + +netcdf and data +``````````````` +The ``in_vars`` section for NetCDF and xarray input acts as a mapping between the variable +names in the input dataset to the variable names expected by MetSim. The format +is given as ``metsim_varname = netcdf_varname``. The minimum required variables +given have ``metsim_varname``\s corresponding to ``t_min``, ``t_max``, and +``prec``; these variable names correspond to minimum daily temperature (Celcius), +maximum daily temperature (Celcius), and precipitation (mm/day). + +ascii +````` +The ``in_vars`` section for ASCII input acts similarly to the NetCDF input +format, except for one key point. Variables should be given as a tautology: the +format is given as ``metsim_varname = metsim_varname``. The order that the +variables are given corresponds to the column numbers that they appear in the +input files. The minimum required variables are ``t_min``, ``t_max``, and +``prec``; these variable names correspond to minimum daily temperature (Celcius), +maximum daily temperature (Celcius), and precipitation (mm/day). + +binary +`````` +This section has an input style for binary files that mimics the VIC version 4 +input style. Each line is specified as ``varname = scale cdatatype``, where +``varname`` is the name that MetSim should use for the column, ``scale`` is a +floating point scaling factor that should be applied after conversion from +binary to floating point; the conversion applied by the ``scale`` is applied +after the value in the input is converted from binary to the ``cdatatype`` +specified for each variable. Valid ``cdatatype``\s are ``signed`` and +``unsigned``. ``signed`` values are interpreted as values which can be positive +or negative, whereas ``unsigned`` values are interpreted as values that can only +be greater than or equal to zero. + +domain_vars section +------------------- +The ``domain_vars`` section is where information about the domain file is given. +Since the domain file is given as a NetCDF file this section has a similar +format to that of the NetCDF input file format described above. That is, +entries should be of the form ``metsim_varname = netcdfvarname``. The minimum +required variables have ``metsim_varname``\s corresponding to ``lat``, ``lon``, +``mask``, and ``elev``; these variable names correspond to latitude, longitude, +a mask of valid cells in the domain, and the elevation given in meters. If +``prec_type`` = ``triangle`` or ``mix``, two additonal variables are required +including ``dur`` and ``t_pk`` for disaggregating daily precipitation according +to the "triangle" method. + +out_vars section +---------------- +The ``out_vars`` section is where you can specify the output variables that you +want to include. There are two formats for this section. The first is the old format, +which we provide backwards compatibility for. You simply specify in the top level +``[MetSim]`` section a list of output variables with the names used by MetSim. They +will be written out with the same names used internally. Available options are +dependent on whether daily or subdaily output is being generated. Options for +daily output are: + +- pet +- shortwave +- t_max +- t_min +- tskc + +Options for subdaily output are: + + - prec + - shortwave + - longwave + - temp + - vapor_pressure + - air_pressure + - tskc + - rel_humid + - spec_humid + - wind + +It may also be preferable to write the output with different variable names. +MetSim allows for this in a similar way as the ``forcing_vars`` section. +The syntax for this type of output specification is as follows: + +.. code-block:: ini + [out_vars] + metsim_varname = output_varname + +constant_vars section +------------------- +The ``constant_vars`` section is optional and allows you to set some of the +forcing inputs to a constant value. The specification simply consists of entries +of the form ``metsim_varname = value``, where ``value`` is a number that can be +converted to a double. There can only be one entry per line. If the +``metsim_varname`` corresponds to an entry that is already in the ``forcing_vars`` +section, then the constant value will take precedence. In the current +implementation there must be at least one non-constant entry in ``forcings_vars`` +(i.e. at least one entry that is not also in ``constant_vars``). + +For example: +``wind = 2.0`` +will result in a constant wind field in the output file. In this case ``wind`` +does not need to be specified in the ``forcing_vars`` section. If it was, it +will still be set to a constant value of 2 m/s. + +Similarly: +``t_max = 30 +t_min = 10`` +will result in output with a diurnal cycle in which the temperature varies at +all locations between 10C and 30C. However, all estimation and disaggregation +routines are still evaluated, with constant ``t_max`` and ``t_min`` as input. diff --git a/docs/examples.rst b/docs/examples.rst index 926000b..a0ad5b1 100644 --- a/docs/examples.rst +++ b/docs/examples.rst @@ -7,80 +7,88 @@ Basics ------ Provided in the source are several examples that can help you to get started using MetSim. They are located in the ``examples`` -directory. We will look at the ``example_nc.conf`` file. Its -contents are: - -.. code-block:: ini +directory. For demonstration here is an example YAML configuration file:: # This is an example of an input file for MetSim - [MetSim] - - # Time step in minutes - time_step = 60 - - # Forcings begin here (year/month/day:hour) - start = 1950/1/1 - - # Forcings end at this date (year/month/day) - stop = 1950/1/31 - - # Input specification - forcing = ./tests/data/test.nc - domain = ./tests/data/domain.nc - state = ./tests/data/state_nc.nc - in_fmt = netcdf - - # Output specification - out_dir = ./results - out_prefix = forcing - out_precision = f8 - - [chunks] - lat = 3 - lon = 3 - - [forcing_vars] - prec = Prec - t_max = Tmax - t_min = Tmin - - [state_vars] - prec = prec - t_max = t_max - t_min = t_min - - [domain_vars] - lat = lat - lon = lon - mask = mask - elev = elev - -This is a minimal configuration file for MetSim, and contains 3 sections. The -first section, ``[MetSim]`` describes some basic settings such as the locations -of data and parameters used in calculations. For a complete description of the -input format see the `configuration `_ page. The key things to note in this section -are the options specified under the ``# Input specification`` and ``# Output -specification`` comment headers. The ``forcing`` and ``domain`` options refer -to the two types of required input, and the ``in_format`` and ``out_format`` -options tell MetSim how they should be treated. - -The second and third sections (``[forcing_vars]`` and ``[state_vars]``) describe the variables in the -datasets provided in the ``forcing`` and ``state`` options of the first section. -The left side of the assignment is the name of the variable given -in the ``forcing`` dataset, while the right hand side is the -name the variable should be given within MetSim. Note that the -variables shown here are the minimum required set to run the -forcing generation. The names given on the right hand side are -also important to name correctly, as they are referenced internally. -If you are unsure what variable names are used internally see the -`configuration `_ page for a full breakdown. + # Overall configuration, specification of parameters and input/output + # paths goes in the "MetSim" section + MetSim: + # Time step in minutes + time_step: 30 + # Forcings begin here (year-month-day) + start: 1950-1-1 + # Forcings end at this date (year-month-day) + stop: 1950-1-31 + # Input and output directories + forcing: './metsim/data/test.nc' + domain: './metsim/data/tiny_domain.nc' + state: './metsim/data/state_nc.nc' + forcing_fmt: 'netcdf' + in_format: 'netcdf' + out_dir: './results' + out_prefix: 'yaml_output' + prec_type: 'triangle' + utc_offset: True + + out_vars: + temp: + out_name: 'airtemp' + units: 'K' + + prec: + out_name: 'pptrate' + units: 'mm/s' + + shortwave: + out_name: 'SWradAtm' + + spec_humid: + out_name: 'spechum' + + air_pressure: + out_name: 'airpres' + units: 'kPa' + + wind: + out_name: 'windspd' + + chunks: + lat: 3 + lon: 3 + + forcing_vars: + # Format is metsim_name: input_name + prec : 'Prec' + t_max : 'Tmax' + t_min : 'Tmin' + + state_vars: + # Format is metsim_name: input_name + prec : 'Prec' + t_max : 'Tmax' + t_min : 'Tmin' + + domain_vars: + # Format is metsim_name: input_name + lat : 'lat' + lon : 'lon' + mask : 'mask' + elev : 'elev' + t_pk : 't_pk' + dur : 'dur' + + constant_vars: + wind : 2.0 + +This is a minimal configuration file for MetSim. For a complete description of the +input format see the `configuration `_ page. To run this example from the command line, once you have installed MetSim, use the following command: -``ms path/to/example_nc.conf --verbose`` +``ms path/to/example.yaml --verbose`` -This will run MetSim and disaggregate to hourly data, and write +This will run MetSim and disaggregate to half-hourly data, and write out the results in a NetCDF file located in the directory specified under ``out_dir`` in the configuration file (here ``./results``). The addition of the ``--verbose`` flag provides some @@ -95,19 +103,3 @@ Daily values can be output by specifying a ``time_step`` of ``1440`` in the configuration file, such as the one shown in the previous section. This will prevent MetSim's disaggregation routines from being run, and the results written out will be daily values. - -Translating formats of daily values ------------------------------------ - -.. warning:: This section `only` applies to daily input and output. - -This section can be useful if you are interested in converting VIC format binary -or ASCII forcing inputs into NetCDF format. - -To configure this behavior, several configuration options will have to be set. -First, the ``time_step`` variable must be set to ``1440`` to enable daily output. -Then, the ``forcing_fmt`` and ``out_fmt`` variables should be specified. The final -option that should be set is ``out_vars``. This can be set to include only the -variables you have in your input file, if you wish to not generate any new data, -or it can be set to include any of the variables generated by the simulator -specified in the ``method`` option. diff --git a/docs/index.rst b/docs/index.rst index bcc1789..9eb5e44 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -107,8 +107,8 @@ in the configuration files see the `configuration `_ page. Once installed, MetSim can be used from the command line via: -.. code-block:: bash - +.. code-block:: bash + usage: ms [-h] [-n NUM_WORKERS] [-s SCHEDULER] [-v] [--version] config positional arguments: @@ -180,5 +180,6 @@ Sitemap examples data configuration + configuration_ini api whats-new diff --git a/docs/whats-new.rst b/docs/whats-new.rst index 65a8ad2..b2f78c4 100644 --- a/docs/whats-new.rst +++ b/docs/whats-new.rst @@ -3,6 +3,22 @@ What's New ========== +.. _whats-new.2.3.0: + +v2.3.0 +------ +Enchancements +~~~~~~~~~~~~~ +- Allow for variable renaming in INI configuration files. +- Added capability for new YAML configuration file format +- Added capability simple unit conversions on output when + using the YAML configuration file format + +Bug fixes +~~~~~~~~~ +- Fixed a bug where `utc_offset` causes radiation to be + incorrectly scaled + .. _whats-new.2.2.0: v2.2.2 diff --git a/examples/example_yaml.yaml b/examples/example_yaml.yaml new file mode 100644 index 0000000..a895b8c --- /dev/null +++ b/examples/example_yaml.yaml @@ -0,0 +1,65 @@ +# This is an example of an input file for MetSim +# Overall configuration, specification of parameters and input/output +# paths goes in the "MetSim" section +MetSim: + # Time step in minutes + time_step: 30 + # Forcings begin here (year-month-day) + start: 1950-1-1 + # Forcings end at this date (year-month-day) + stop: 1950-1-31 + # Input and output directories + forcing: './metsim/data/test.nc' + domain: './metsim/data/tiny_domain.nc' + state: './metsim/data/state_nc.nc' + forcing_fmt: 'netcdf' + in_format: 'netcdf' + out_dir: './results' + out_prefix: 'yaml_output' + prec_type: 'triangle' + utc_offset: True + +out_vars: + temp: + out_name: 'airtemp' + units: 'K' + prec: + out_name: 'pptrate' + units: 'mm s-1' + shortwave: + out_name: 'SWradAtm' + spec_humid: + out_name: 'spechum' + air_pressure: + out_name: 'airpres' + units: 'kPa' + wind: + out_name: 'windspd' + +chunks: + lat: 3 + lon: 3 + +forcing_vars: + # Format is metsim_name: input_name + prec : 'Prec' + t_max : 'Tmax' + t_min : 'Tmin' + +state_vars: + # Format is metsim_name: input_name + prec : 'prec' + t_max : 't_max' + t_min : 't_min' + +domain_vars: + # Format is metsim_name: input_name + lat : 'lat' + lon : 'lon' + mask : 'mask' + elev : 'elev' + t_pk : 't_pk' + dur : 'dur' + +constant_vars: + wind : 2.0 diff --git a/metsim/cli/ms.py b/metsim/cli/ms.py index c50b5e2..ffb3ff1 100755 --- a/metsim/cli/ms.py +++ b/metsim/cli/ms.py @@ -26,6 +26,7 @@ import sys from collections import OrderedDict from configparser import ConfigParser +from metsim import __name__, __version__, io def _is_valid_file(parser, arg): @@ -38,7 +39,6 @@ def _is_valid_file(parser, arg): def parse(args): """Parse the command line arguments""" - from metsim import __name__, __version__ parser = argparse.ArgumentParser() parser.add_argument('config', type=lambda x: _is_valid_file(parser, x), help='Input configuration file') @@ -54,61 +54,10 @@ def parse(args): return parser.parse_args(args) -def init(opts): - """Initialize some information based on the options & config""" - config = ConfigParser() - config.optionxform = str - config.read(opts.config) - conf = OrderedDict(config['MetSim']) - - def invert_dict(d): - return OrderedDict([reversed(item) for item in d.items()]) - - def to_list(s): - return json.loads(s.replace("'", '"').split('#')[0]) - - conf['forcing_vars'] = OrderedDict(config['forcing_vars']) - if conf['forcing_fmt'] != 'binary': - conf['forcing_vars'] = invert_dict(conf['forcing_vars']) - conf['domain_vars'] = invert_dict(OrderedDict(config['domain_vars'])) - conf['state_vars'] = invert_dict(OrderedDict(config['state_vars'])) - conf['chunks'] = OrderedDict(config['chunks']) - if 'constant_vars' in config: - conf['constant_vars'] = OrderedDict(config['constant_vars']) - - # If the forcing variable is a directory, scan it for files - if os.path.isdir(conf['forcing']): - forcing_files = [os.path.join(conf['forcing'], fn) for fn in - next(os.walk(conf['forcing']))[2]] - else: - forcing_files = conf['forcing'] - - # Ensure that parameters with boolean values are correctly recorded - for bool_param in ['utc_offset', 'period_ending']: - if (bool_param in conf.keys() - and conf[bool_param].strip().lower() == 'true'): - conf[bool_param] = True - else: - conf[bool_param] = False - - # Update the full configuration - conf.update({"calendar": conf.get('calendar', 'standard'), - "scheduler": opts.scheduler, - "num_workers": opts.num_workers, - "verbose": logging.DEBUG if opts.verbose else logging.INFO, - "forcing": forcing_files, - "out_dir": os.path.abspath(conf['out_dir']), - "prec_type": conf.get('prec_type', 'uniform')}) - conf['out_vars'] = to_list(conf.get('out_vars', '[]')) - - conf = {k: v for k, v in conf.items() if v != []} - return conf - - def main(): """Runs MetSim""" from metsim.metsim import MetSim - setup = init(parse(sys.argv[1:])) + setup = io.read_config(parse(sys.argv[1:])) ms = MetSim(setup) ms.run() diff --git a/metsim/data/tiny_domain.nc b/metsim/data/tiny_domain.nc new file mode 100644 index 0000000..28f7eba Binary files /dev/null and b/metsim/data/tiny_domain.nc differ diff --git a/metsim/disaggregate.py b/metsim/disaggregate.py index d226da3..85bef72 100644 --- a/metsim/disaggregate.py +++ b/metsim/disaggregate.py @@ -88,6 +88,9 @@ def disaggregate(df_daily: pd.DataFrame, params: dict, df_disagg['vapor_pressure'] = vapor_pressure( df_daily['vapor_pressure'].values, df_disagg['temp'].values, t_Tmin, n_disagg, ts) + df_disagg['vapor_pressure'] = (df_disagg['vapor_pressure'] + .fillna(method='ffill') + .fillna(method='bfill')) df_disagg['rel_humid'] = relative_humidity( df_disagg['vapor_pressure'].values, df_disagg['temp'].values) @@ -694,19 +697,31 @@ def shortwave(sw_rad: np.array, daylength: np.array, day_of_year: np.array, ts_per_day = int(cnst.HOURS_PER_DAY * cnst.MIN_PER_HOUR / ts) disaggrad = np.zeros(int(n_days * ts_per_day)) rad_fract_per_day = int(cnst.SEC_PER_DAY / cnst.SW_RAD_DT) + tmp_rad = np.repeat(tmp_rad, rad_fract_per_day) if params['utc_offset']: - utc_offset = int(((params.get("lon", 0) - params.get("theta_s", 0)) / - cnst.DEG_PER_REV) * rad_fract_per_day) + utc_offset = int((params['lon'] / cnst.DEG_PER_REV) * rad_fract_per_day) tiny_rad_fract = np.roll(tiny_rad_fract.flatten(), -utc_offset) + tmp_rad = np.roll(tmp_rad.flatten(), -utc_offset) + tiny_rad_fract = tiny_rad_fract.flatten() else: + utc_offset = 0 tiny_rad_fract = tiny_rad_fract.flatten() chunk_size = int(ts * (cnst.SEC_PER_MIN / cnst.SW_RAD_DT)) ts_id = np.repeat(np.arange(ts_per_day), chunk_size) for day in range(n_days): + # Mask to select out from tiny_rad_fract radslice = slice((day_of_year[day] - 1) * rad_fract_per_day, (day_of_year[day]) * rad_fract_per_day) rad = tiny_rad_fract[radslice] + + # Mask to select out time chunk to place disaggregated values into dslice = slice(int(day * ts_per_day), int((day + 1) * ts_per_day)) - rad_chunk = np.bincount(ts_id, weights=rad) - disaggrad[dslice] = rad_chunk * tmp_rad[day] + + # Mask to weight daily solar radiation with + weight_slice = slice(int(day * rad_fract_per_day), + int((day + 1) * rad_fract_per_day)) + + rad_chunk = np.bincount(ts_id, weights=rad * tmp_rad[weight_slice]) + disaggrad[dslice] = rad_chunk + return disaggrad diff --git a/metsim/io.py b/metsim/io.py index ba99bd8..381f5c2 100644 --- a/metsim/io.py +++ b/metsim/io.py @@ -21,12 +21,147 @@ import os import struct import logging +import json +import yaml +import warnings import numpy as np import pandas as pd import xarray as xr +from collections import OrderedDict +from configparser import ConfigParser from metsim.datetime import date_range +from metsim import metsim + + +def _invert_dict(d): + return OrderedDict([reversed(item) for item in d.items()]) + + +def _to_list(s): + return json.loads(s.replace("'", '"').split('#')[0]) + + +def check_config_type(config_file): + # Search for first non-comment line + line = '#' + with open(config_file, 'r') as f: + while line.strip().startswith('#'): + line = f.readline() + # Strip out any inline comment + line = line.split('#')[0].strip() + if line == '[MetSim]': + return 'ini' + if line == 'MetSim:': + return 'yaml' + # fallback + return 'ini' + + +def read_config(opts): + """Initialize some information based on the options & config""" + config = ConfigParser() + config.optionxform = str + config_file = opts.config + config_type = check_config_type(config_file) + config_readers = {'yaml': read_yaml_config, + 'ini': read_ini_config} + return config_readers[config_type](config_file, opts) + + +def read_yaml_config(config_file, opts): + with open(config_file, 'r') as f: + config = yaml.load(f, Loader=yaml.SafeLoader) + conf = OrderedDict(config['MetSim']) + conf['forcing_vars'] = _invert_dict(OrderedDict(config['forcing_vars'])) + conf['domain_vars'] = _invert_dict(OrderedDict(config['domain_vars'])) + conf['state_vars'] = _invert_dict(OrderedDict(config['state_vars'])) + conf['out_vars'] = OrderedDict(config['out_vars']) + for k, v in conf['out_vars'].items(): + if 'units' not in v: + v['units'] = metsim.available_outputs[k]['units'] + conf['chunks'] = OrderedDict(config['chunks']) + if 'constant_vars' in config: + conf['constant_vars'] = OrderedDict(config['constant_vars']) + + # If the forcing variable is a directory, scan it for files + if os.path.isdir(conf['forcing']): + forcing_files = [os.path.join(conf['forcing'], fn) for fn in + next(os.walk(conf['forcing']))[2]] + else: + forcing_files = conf['forcing'] + + # Update the full configuration + conf.update({"calendar": conf.get('calendar', 'standard'), + "scheduler": opts.scheduler, + "num_workers": opts.num_workers, + "verbose": logging.DEBUG if opts.verbose else logging.INFO, + "forcing": forcing_files, + "out_dir": os.path.abspath(conf['out_dir'])}) + + conf = {k: v for k, v in conf.items() if v != []} + return conf + + +def read_ini_config(config_file, opts): + config = ConfigParser() + config.optionxform = str + config.read(config_file) + conf = OrderedDict(config['MetSim']) + conf['forcing_vars'] = OrderedDict(config['forcing_vars']) + if conf['forcing_fmt'] != 'binary': + conf['forcing_vars'] = _invert_dict(conf['forcing_vars']) + conf['domain_vars'] = _invert_dict(OrderedDict(config['domain_vars'])) + conf['state_vars'] = _invert_dict(OrderedDict(config['state_vars'])) + conf['chunks'] = OrderedDict(config['chunks']) + if 'constant_vars' in config: + conf['constant_vars'] = OrderedDict(config['constant_vars']) + + # If the forcing variable is a directory, scan it for files + if os.path.isdir(conf['forcing']): + forcing_files = [os.path.join(conf['forcing'], fn) for fn in + next(os.walk(conf['forcing']))[2]] + else: + forcing_files = conf['forcing'] + + # Ensure that parameters with boolean values are correctly recorded + for bool_param in ['utc_offset', 'period_ending']: + if (bool_param in conf.keys() + and conf[bool_param].strip().lower() == 'true'): + conf[bool_param] = True + else: + conf[bool_param] = False + + # Update the full configuration + conf.update({"calendar": conf.get('calendar', 'standard'), + "scheduler": opts.scheduler, + "num_workers": opts.num_workers, + "verbose": logging.DEBUG if opts.verbose else logging.INFO, + "forcing": forcing_files, + "out_dir": os.path.abspath(conf['out_dir']), + "prec_type": conf.get('prec_type', 'uniform')}) + + # List variant + if 'out_vars' in conf: + conf['out_vars'] = _to_list(conf['out_vars']) + temp = {} + for ov in conf['out_vars']: + temp[ov] = metsim.available_outputs[ov] + conf['out_vars'] = temp + else: + conf['out_vars'] = {} + # Dict variant + if 'out_vars' in config: + temp = {} + for varname, outname in config['out_vars'].items(): + temp[varname] = metsim.available_outputs[varname] + temp[varname]['out_name'] = outname + conf['out_vars'].update(temp) + + conf = {k: v for k, v in conf.items() if v != []} + return conf + def read_met_data(params: dict, domain: xr.Dataset) -> xr.Dataset: diff --git a/metsim/metsim.py b/metsim/metsim.py index 20b3212..e76518b 100644 --- a/metsim/metsim.py +++ b/metsim/metsim.py @@ -33,6 +33,7 @@ import logging import os import sys +import warnings import time as tm from collections import Iterable, OrderedDict from getpass import getuser @@ -54,6 +55,7 @@ from metsim.disaggregate import disaggregate from metsim.methods import mtclim from metsim.physics import solar_geom +from metsim.units import converters NO_SLICE = {} DASK_CORE_SCHEDULERS = ['multiprocessing', 'threading', 'synchronous', @@ -102,21 +104,24 @@ 'rel_humid': {'units': '%', 'long_name': 'relative humidity', 'standard_name': 'relative_humidity', 'missing_value': np.nan, 'fill_value': np.nan}, - 'spec_humid': {'units': '', 'long_name': 'specific humidity', + 'spec_humid': {'units': 'g g-1', 'long_name': 'specific humidity', 'standard_name': 'specific_humidity', 'missing_value': np.nan, 'fill_value': np.nan}, - 'wind': {'units': 'm/s', 'long_name': 'wind speed', + 'wind': {'units': 'm s-1', 'long_name': 'wind speed', 'standard_name': 'wind_speed', 'missing_value': np.nan, 'fill_value': np.nan}, '_global': {'conventions': '1.6', 'title': 'Output from MetSim', 'institution': 'University of Washington', 'source': 'metsim.py', 'history': 'Created: {0} by {1}'.format(now, user), - 'references': references, 'comment': 'no comment at this time'}} attrs = {k: OrderedDict(v) for k, v in attrs.items()} +available_outputs = {n : {'out_name': n, 'units': attrs[n]['units']} + for n in attrs if n != '_global'} +default_outputs = ['temp', 'prec', 'shortwave', 'longwave', + 'vapor_pressure', 'rel_humid'] class MetSim(object): """ @@ -137,6 +142,7 @@ class MetSim(object): "out_prefix": 'forcing', "start": 'forcing', "stop": 'forcing', + "forcing_fmt": 'netcdf', "time_step": -1, "calendar": 'standard', "prec_type": 'uniform', @@ -151,8 +157,7 @@ class MetSim(object): "rain_scalar": 0.75, "tday_coef": 0.45, "lapse_rate": 0.0065, - "out_vars": ['temp', 'prec', 'shortwave', 'longwave', - 'vapor_pressure', 'rel_humid'], + "out_vars": {n: available_outputs[n] for n in default_outputs}, "out_freq": None, "chunks": NO_SLICE, "scheduler": 'distributed', @@ -213,6 +218,22 @@ def __init__(self, params: dict, domain_slice=NO_SLICE): freq=self.params['out_freq'], period_ending=self.params['period_ending']) + self._update_unit_attrs(self.params['out_vars']) + + def _update_unit_attrs(self, out_vars): + for k, v in out_vars.items(): + if 'units' in v.keys(): + if v['units'] in converters[k].keys(): + attrs[k]['units'] = v['units'] + else: + self.logger.warn( + f'Could not find unit conversion for {k} to {v["units"]}!' + f' We will use the default units of' + f' {available_outputs[k]["units"]} instead.' ) + v['units'] = available_outputs[k]['units'] + else: + v['units'] = available_outputs[k]['units'] + def _validate_force_times(self, force_times): for p, i in [('start', 0), ('stop', -1)]: @@ -343,9 +364,9 @@ def setup_netcdf_output(self, filename, times): for d, size in zip(var_dims, dim_sizes): ncout.createDimension(d, size) # vars - for varname in self.params['out_vars']: + for varname, varconf in self.params['out_vars'].items(): ncout.createVariable( - varname, self.params['out_precision'], var_dims, + varconf['out_name'], self.params['out_precision'], var_dims, **create_kwargs) # add metadata and coordinate variables (time/lat/lon) @@ -365,10 +386,14 @@ def setup_netcdf_output(self, filename, times): dim_var = ncout.createVariable(dim, dim_dtype, (dim, )) dim_var[:] = dim_vals - for p in ['elev', 'lat', 'lon', 'is_worker']: - if p in self.params: - self.params.pop(p) + # parameters to not record in the metadata + skip_params = ['elev', 'lat', 'lon', 'is_worker', 'out_vars', + 'forcing_vars', 'domain_vars', 'state_vars', + 'constant_vars', 'references', 'verbose', + 'num_workers',] for k, v in self.params.items(): + if k in skip_params: + continue # Need to convert some parameters to strings if k in ['start', 'stop', 'utc_offset', 'period_ending']: v = str(v) @@ -390,9 +415,12 @@ def setup_netcdf_output(self, filename, times): setattr(ncout, key, val) # set variable attrs - for varname in ncout.variables: + for key, value in attrs.get('time', {}).items(): + setattr(ncout.variables['time'], key, value) + for varname, varconf in self.params['out_vars'].items(): + outname = varconf['out_name'] for key, val in attrs.get(varname, {}).items(): - setattr(ncout.variables[varname], key, val) + setattr(ncout.variables[outname], key, val) def write_chunk(self, locks=None): '''write data from a single chunk''' @@ -404,11 +432,12 @@ def write_chunk(self, locks=None): time_slice = slice(times[0], times[-1]) with lock: with Dataset(filename, mode="r+") as ncout: - for varname in self.params['out_vars']: - dims = ncout.variables[varname].dimensions[1:] + for varname, varconf in self.params['out_vars'].items(): + outname = varconf['out_name'] + dims = ncout.variables[outname].dimensions[1:] write_slice = ((slice(None), ) + tuple( self._domain_slice[d] for d in dims)) - ncout.variables[varname][write_slice] = ( + ncout.variables[outname][write_slice] = ( self.output[varname].sel(time=time_slice).values) def run_slice(self): @@ -446,8 +475,12 @@ def run_slice(self): self.disagg, times) # Cut the returned data down to the correct time index + # and do any required unit conversions for varname in self.params['out_vars']: - self.output[varname][locs] = df[varname].values + desired_units = self.params['out_vars'][varname]['units'] + out_vals = converters[varname][desired_units]( + df[varname].values, int(self.params['time_step'])) + self.output[varname][locs] = out_vals def _unpack_state(self, result: pd.DataFrame, locs: dict): """Put restart values in the state dataset""" @@ -707,13 +740,8 @@ def wrap_run_cell(func: callable, params: dict, # chunks - if no data is available, just repeat some # default values (this case is used at the very # beginning and end of the record) - try: - prevday = out_times[0] - pd.Timedelta('1 days') - t_begin = [ds['t_min'].sel(time=prevday), - ds['t_max'].sel(time=prevday)] - except (KeyError, ValueError): - t_begin = [state['t_min'].values[-1], - state['t_max'].values[-1]] + t_begin = [state['t_min'].values[-1], + state['t_max'].values[-1]] try: nextday = out_times[-1] + pd.Timedelta('1 days') t_end = [ds['t_min'].sel(time=nextday), diff --git a/metsim/tests/test_metsim.py b/metsim/tests/test_metsim.py index aad1a52..2dab1fa 100755 --- a/metsim/tests/test_metsim.py +++ b/metsim/tests/test_metsim.py @@ -14,7 +14,9 @@ import xarray as xr import metsim.cli.ms as cli +import metsim.metsim from metsim.metsim import MetSim +from metsim import io class DummyOpts: @@ -226,7 +228,8 @@ def test_time_offset(): 'time_step': "60", 'out_dir': out_dir, 'out_state': os.path.join(out_dir, 'state.nc'), - 'out_vars': out_vars, + 'out_vars': {n: metsim.metsim.available_outputs[n] + for n in out_vars}, 'forcing_vars': in_vars_section['binary'], 'domain_vars': domain_section['binary'] } @@ -243,6 +246,91 @@ def test_time_offset(): assert ms1._times[1:] == ms2._times[:-1] +def test_variable_rename(): + """Tests to make sure that variable renaming works""" + loc = data_locations['binary'] + data_files = [os.path.join(loc, f) for f in os.listdir(loc)] + out_dir = '.' + params = {'start': dates['binary'][0], + 'stop': dates['binary'][1], + 'forcing_fmt': 'binary', + 'domain_fmt': 'netcdf', + 'state_fmt': 'netcdf', + 'domain': './metsim/data/stehekin.nc', + 'state': './metsim/data/state_vic.nc', + 'forcing': data_files, + 'method': 'mtclim', + 'scheduler': 'threading', + 'time_step': "60", + 'out_dir': out_dir, + 'out_state': os.path.join(out_dir, 'state.nc'), + 'out_vars': { + 'prec': {'out_name': 'pptrate'}, + 'shortwave': {'out_name': 'SWRadAtm'}}, + 'forcing_vars': in_vars_section['binary'], + 'domain_vars': domain_section['binary'] + } + ms = MetSim(params) + ms.run() + ds = ms.open_output() + assert 'pptrate' in ds.variables + assert 'SWRadAtm' in ds.variables + + +def test_unit_conversion(): + """Tests to make sure that variable renaming works""" + loc = data_locations['binary'] + data_files = [os.path.join(loc, f) for f in os.listdir(loc)] + out_dir = '.' + params = {'start': dates['binary'][0], + 'stop': dates['binary'][1], + 'forcing_fmt': 'binary', + 'domain_fmt': 'netcdf', + 'state_fmt': 'netcdf', + 'domain': './metsim/data/stehekin.nc', + 'state': './metsim/data/state_vic.nc', + 'forcing': data_files, + 'method': 'mtclim', + 'scheduler': 'threading', + 'time_step': "60", + 'out_dir': out_dir, + 'out_state': os.path.join(out_dir, 'state.nc'), + 'out_vars': { + 'prec': {'out_name': 'pptrate', + 'units': 'mm s-1'}, + 'temp': {'out_name': 'airtemp', + 'units': 'K'}}, + 'forcing_vars': in_vars_section['binary'], + 'domain_vars': domain_section['binary']} + + params1 = dict() + params1.update(params) + params2 = dict() + params2.update(params) + params2['out_vars'] = { + 'prec': {'out_name': 'pptrate', + 'units': 'mm timestep-1'}, + 'temp': {'out_name': 'airtemp', + 'units': 'C'}} + ms1 = MetSim(params1) + ms1.run() + ds1 = ms1.open_output().load() + ds1.close() + time_step = int(params['time_step']) + sec_per_min = 60. + tol = 1e-4 + + ms2 = MetSim(params2) + ms2.run() + ds2 = ms2.open_output().load() + + assert np.allclose(ds1['airtemp'].mean(), + ds2['airtemp'].mean()+273.15, atol=tol) + assert np.allclose(time_step * sec_per_min * ds1['pptrate'].mean(), + ds2['pptrate'].mean(), atol=tol) + + + def test_disaggregation_values(): """Tests to make sure values are being generated correctly""" # Set parameters @@ -264,7 +352,8 @@ def test_disaggregation_values(): 'time_step': "60", 'out_dir': out_dir, 'out_state': os.path.join(out_dir, 'state.nc'), - 'out_vars': out_vars, + 'out_vars': {n: metsim.metsim.available_outputs[n] + for n in out_vars}, 'forcing_vars': in_vars_section['binary'], 'domain_vars': domain_section['binary'] } @@ -273,7 +362,7 @@ def test_disaggregation_values(): def check_data(out, good, tol=0.03): assert isinstance(out, pd.DataFrame) - for var in ms.params['out_vars']: + for var in ms.params['out_vars'].keys(): # Check to make sure each variable has normalized # rmse of less than 0.02 h = max([good[var].max(), out[var].max()]) @@ -322,7 +411,7 @@ def test_coordinate_dimension_matchup(): latitude='lat', longitude='lon', mask='mask', elevation='elev', pptrate='prec', maxtemp='t_max', mintemp='t_min') filename = './examples/example_dimtest.conf' - conf = cli.init(DummyOpts(filename)) + conf = io.read_config(DummyOpts(filename)) conf['out_dir'] = tempfile.mkdtemp('results') ms = MetSim(conf) ds = xr.open_dataset('./metsim/data/dim_test.nc') @@ -336,9 +425,21 @@ def test_coordinate_dimension_matchup(): 'constant_vars_nc']) def test_examples(kind): filename = './examples/example_{kind}.conf'.format(kind=kind) - conf = cli.init(DummyOpts(filename)) + conf = io.read_config(DummyOpts(filename)) out_dir = tempfile.mkdtemp('results') conf['out_dir'] = out_dir ms = MetSim(conf) ms.run() assert ms.open_output() is not None + + +def test_yaml_config(): + filename = './examples/example_yaml.yaml' + conf = io.read_config(DummyOpts(filename)) + out_dir = tempfile.mkdtemp('results') + conf['out_dir'] = out_dir + ms = MetSim(conf) + ms.run() + assert ms.open_output() is not None + + diff --git a/metsim/units.py b/metsim/units.py new file mode 100644 index 0000000..8f985f9 --- /dev/null +++ b/metsim/units.py @@ -0,0 +1,56 @@ +import metsim.constants as cnst + +converters = { + 'pet': { + 'mm timestep-1': lambda x, ts: x, + 'mm h-1': lambda x, ts: x / (ts / cnst.MIN_PER_HOUR), + 'mm s-1': lambda x, ts: x / (ts * cnst.SEC_PER_MIN), + }, + 'prec': { + 'mm timestep-1': lambda x, ts: x, + 'mm h-1': lambda x, ts: x / (ts / cnst.MIN_PER_HOUR), + 'mm s-1': lambda x, ts: x / (ts * cnst.SEC_PER_MIN), + }, + 'shortwave': { + 'W m-2': lambda x, ts: x, + }, + 'longwave': { + 'W m-2': lambda x, ts: x, + }, + 't_max': { + 'C': lambda x, ts: x, + 'K': lambda x, ts: x + 273.15, + }, + 't_min': { + 'C': lambda x, ts: x, + 'K': lambda x, ts: x + 273.15, + }, + 'temp': { + 'C': lambda x, ts: x, + 'K': lambda x, ts: x + 273.15, + }, + 'vapor_pressure': { + 'kPa': lambda x, ts: x, + 'hPa': lambda x, ts: x * 100., + 'Pa': lambda x, ts: x * 1000., + }, + 'air_pressure': { + 'kPa': lambda x, ts: x, + 'hPa': lambda x, ts: x * 100., + 'Pa': lambda x, ts: x * 1000., + }, + 'tskc': { + 'fraction': lambda x, ts: x, + '%': lambda x, ts: x * 100., + }, + 'rel_humid': { + '%': lambda x, ts: x, + 'fraction': lambda x, ts: x / 100., + }, + 'spec_humid': { + 'g g-1': lambda x, ts: x, + }, + 'wind': { + 'm s-1': lambda x, ts: x, + } + }