Commit messages are missing

[kaskranenburgQ]

In some datasets, the commit messages of the data can be missing in the frontend, while these can be found in the commit messages file.
It seems this occurs in the following cases:

• European datasets: Housing stock & Typical heat demand data.
• Most data points for Switzerland, Norway and Serbia. Some data points are filled in with "Value sourced from the country's energy balance", this is not a commit message that can be found in the commit file for these countries. This needs some investigation as well.

Notifying @mabijkerk, will look into this myself.

[kndehaan]

Some data points are filled in with "Value sourced from the country's energy balance", this is not a commit message that can be found in the commit file for these countries.

This is a default commit message, defined in en.yml and nl.yml, used when inputs for entso datasets are retrieved from the country's energy balance.

I looked at how the commit messages were defined in different migrations on ETLocal. It seems that it goes wrong when multiple fields have the same commit message, for example in 20241106112225_eu_datasets_building_stock_update, where the following setup is used:

- fields:
  - typical_useful_demand_for_space_heating_semi_detached_houses_future
  - typical_useful_demand_for_space_heating_semi_detached_houses_2005_present
  - typical_useful_demand_for_space_heating_semi_detached_houses_1985_2004
  - typical_useful_demand_for_space_heating_semi_detached_houses_1965_1984
  - typical_useful_demand_for_space_heating_semi_detached_houses_1945_1964
  - typical_useful_demand_for_space_heating_semi_detached_houses_before_1945
  message:
    Derived from the Hotmaps project (2018), overview of building age and energy demand of all EU countries,
    see ...

This commit message is not visible in the frontend for the respective inputs.

When taking the PV27_2022 (Noord-Holland) dataset as an example, the commits file uses the following setup:

- fields: 
  - input_industry_chemical_fertilizers_electricity_demand
  - input_industry_chemical_fertilizers_network_gas_demand
  message:
   "Nagenoeg geen grootschalige kunstmestproductie in Noord-Holland. Volgens emissieregistratie bevindt 0.4% van de uitstoot uit de kunstmestproductie zich in Noord-Holland. Energiedata niet bekend. "

This commit message does show up in the fronted.

The difference here is that the commit message is stated between " ", perhaps this would solve the issue when multiple fields have the same commit message? I'm not sure if this would solve it but it might be worth a try...

@kaskranenburgQ @mabijkerk what is the desired result of addressing this issue? We could check all / some (?) datasets and retrospectively try to make these commit messages visible through a new migration. But for which datasets do we want this? I could see that from the EU datasets, it would make sense to check if all commit messages came through from the technology split update and building stock update (20241119104514_eu_households_technology_splits and 20241106112225_eu_datasets_building_stock_update). What are your thought on this?

[github-actions]

This issue has had no activity for 60 days and will be closed in 7 days. Removing the "Stale" label or posting a comment will prevent it from being closed automatically. You can also add the "Pinned" label to ensure it isn't marked as stale in the future.

[mabijkerk]

It seems more like a dev issue right @kaskranenburgQ?