What is the appropriate way to reference other "typed" variables in a config yaml?

[mkavulich]

I am attempting to read in a config file that references other values, some of which are typed with yaml tags such as !!timestamp. However, I can not get the behavior I desire, and the error messages I'm getting are obscure and deep in the uwtools code.

Example code

This is a minimum reproducible example code:

import uwtools.api.config as uwconfig
  
cfg_d = uwconfig.get_yaml_config('test.yaml')
cfg_d.dereference()
print(cfg_d)

And here is the test.yaml file I am attempting to read in:

hafs:
  DATE_FIRST_CYCL: !!timestamp '1900-01-01 12:00:00'
  DATE_LAST_CYCL: '{{ hafs.DATE_FIRST_CYCL }}'

So what I want is two python date objects from these two variables, but I can not manage to make that happen. With this first example, DATE_LAST_CYCL ends up as a string, not a datetime object

$ python test_get_yaml_config.py 
hafs:
  DATE_FIRST_CYCL: 1900-01-01 12:00:00
  DATE_LAST_CYCL: '1900-01-01 12:00:00'

If I change my test.yaml to include the !!timestamp tag (DATE_LAST_CYCL: !!timestamp '{{ hafs.DATE_FIRST_CYCL }}'), I get an obscure error:

$ python test_get_yaml_config.py
Traceback (most recent call last):
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/test_get_yaml_config.py", line 3, in <module>
    cfg_d = uwconfig.get_yaml_config('test.yaml')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/uwtools/api/config.py", line 109, in get_yaml_config
    return _YAMLConfig(config=_ensure_data_source(_str2path(config), stdin_ok))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/uwtools/config/formats/base.py", line 38, in __init__
    self.data = self._load(self._config_file)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/uwtools/config/formats/yaml.py", line 77, in _load
    config = yaml.load(f.read(), Loader=loader)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/yaml/constructor.py", line 51, in get_single_data
    return self.construct_document(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/yaml/constructor.py", line 60, in construct_document
    for dummy in generator:
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/yaml/constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/BMC/hmtb/HAFS/add_vx_workflow/HAFS/ush/conda/envs/hafs_vx/lib/python3.12/site-packages/yaml/constructor.py", line 325, in construct_yaml_timestamp
    values = match.groupdict()
             ^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'groupdict'

My questions

1. Is there a way to do what I'm trying to do?
2. Can the get_yaml_config function be cleaned up to handle this error condition in a less confusing way?

Thanks, and let me know if you need more info.

[christinaholtNOAA]

@mkavulich Thanks for bringing this use case to our attention. It seems like we have an ordering problem here with when pyyaml interacts with the tag and when the Jinja template is filled in. We have a couple of other tags that we can use to help navigate this issue, like !int and !float. I'll look into adding a !datetime one to see if that helps this use case along.

[christinaholtNOAA]

I have replicated the example you provided and locally modified uwtools to accept a !datetime tag that will convert a Jinja expression to a datetime object after it's dereferenced. During that process, I also tested a theory that pyyaml understands an entry that looks like a date without the !!timestamp tag, and found that it's unnecessary when you don't include quotes. This might make things a little less frustrating when the requirement to use a tag for the Jinja-templated version comes into play.

Here's the behavior I see with my local mod. Let me know if this is what you think might work. My test.yaml contains:

hafs:
  DATE_FIRST_CYCL: 1900-01-01 12:00:00
  DATE_LAST_CYCL: !datetime '{{ hafs.DATE_FIRST_CYCL }}'

I use the CLI to replicate what you've shown in the API above:

 $ uw config realize -i test.yaml --output-format yaml
hafs:
  DATE_FIRST_CYCL: 1900-01-01 12:00:00
  DATE_LAST_CYCL: 1900-01-01 12:00:00

It also works when I leave the !!timestamp flag in where you had it before, but looks a bit more cluttered:

hafs:
  DATE_FIRST_CYCL: !!timestamp "1900-01-01 12:00:00"
  DATE_LAST_CYCL: !datetime '{{ hafs.DATE_FIRST_CYCL }}'

We do the same sort of thing when we reference other integers or float values using Jinja expressions. Check out this example where test.yaml contains:

hafs:
  inta: 2
  intb: !int "{{ hafs.inta }}"

We must use the !int flag to make sure that intb comes out as an integer and not a string.

$ uw config realize -i test.yaml --output-format yaml
hafs:
  inta: 2
  intb: 2

And without the !int tag:

$ uw config realize -i test.yaml --output-format yaml
hafs:
  inta: 2
  intb: '2'

If this behavior makes sense to close this gap, I'll open a PR for it.

[mkavulich]

Thanks for the explanation. I don't have an immediate need for this function, but it would be helpful in the future, so I will look for it in the next release.

[christinaholtNOAA]

This new functionality has been proposed in PR #597 and will be available in our next minor release.