Bidding zones representation + custom busmap

[FabianHofmann]

The following changes add an additional layer for bidding zones on the administrative shapes introduced in #1502.

On top, a cluster mode custom_busshapes was added to support passing user-defined busshapes (instead of custom busmap) which is used to perform the clustering.

Note that this is two distinct features.

TODO

• Crete does not have a substation according to OSM prebuilt. however there should be one, as well as the interconnection between the main land and the island (https://www.admie.gr/en/erga/erga-diasyndeseis/diasyndesi-tis-kritis-me-tin-peloponniso). This missing piece is crashing the party a bit, as create has its own bidding zone.

@bobbyxng @tgilon if you already want to take a look feel free.

Checklist

• I tested my contribution locally and it works as intended.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in config/config.default.yaml.
• Changes in configuration options are documented in doc/configtables/*.csv.
• Sources of newly added data are documented in doc/data_sources.rst.
• A release note doc/release_notes.rst is added.

[tgilon]

Thank you @FabianHofmann! I added some comments. I haven't had the chance to review cluster_network.py yet.

[tgilon]

When creating the administrative shapes from the bidding zones (build_admin_shapes from base_network), SE3 (bidding zone) is modified to match SE213 (NUTS3 zone) in the SE4 (bidding zone). This results in a change of the bidding zone for the bus way/120248140 (in the city of Västervik) from SE3 to SE4.

Activating the clustering/simplify_network/remove_stubs_across_borders solves the problem. Do you see any argument against it? @FabianHofmann

Original bidding zones from Electricity Maps

Västervik is in SE3.

Zone SE213 triggering the switch

Västervik is in SE213.

Bidding zone after base_network (admin_shapes.geojson)

Västervik is in SE4.

Node way/12048140 changing from zone

[FabianHofmann]

great @tgilon that you could track it down! this is totally fine

[FabianHofmann]

@tgilon would you mind adding a release note and checking that configtables documentation (in doc/configtables/) is in line?

[tgilon]

@tgilon would you mind adding a release note and checking that configtables documentation (in doc/configtables/) is in line?

Done, but not pushed yet!

@FabianHofmann I haven't tested the custom_busshapes feature yet. What do you think about keeping it separate from this PR?

[FabianHofmann]

mmh, I would say let's keep it, it is quite convenient. @fneum @bobbyxng does one of you also want to review?

[fneum]

Looks mostly good,

Some documentation adjustments for clustering settings and release notes are yet to be included.

Main points:

• Infinite links in copperplate_buses
• Bidding zone mapping based on points rather than Voronoi overlap
• Optionality of build_bidding_zones

I'll give it a spin now.

[fneum]

As far as I understood, this is for the case that you want one country with bidding zone representation and others in NUTSX, right? I didn't understand why this is not handled directly when creating the busmap. Wouldn't the regular handling of the busmap variable do everything you need?

[tgilon]

I'll make this option available for all the clustering modes. It's not handled in the busmap because we need at least one node per country for the following steps of the workflow. This option enables it.

[tgilon]

And to answer the question, it's not specifically for mixed topologies. The main use will be to copperplate nodes to model market zones.

[fneum]

This function can be simplified to a simple sjoin on the substation coordinates. Similar to the NUTS clustering, we don't want to use the (aggregated) Voronoi shapes for anything. The regions associated with the clustered buses should match the original bidding zone shapes.

[FabianHofmann]

I had the same thought however it was not working as I hoped. the problem is that the borders of bidding zones and nuts regions are not perfectly aligned. which means, that nearly all nuts regions that are at a border of a bidding zone always get assigned to the correct bidding zone and the neighbouring (due to a slight overlap). Roughly, a third of all nuts regions fall into this category. given that we would then need the fallback case anyway, I implemented directly with the intersection logic. performance wise this is no problem, it's still fast.

[fneum]

What role do the NUTS regions play here? Couldn't you just look at what substations/buses in the base.nc are within the bidding zone shapefiles? No need to involve NUTS shapes in this case, right?

[tgilon]

The idea discussed with @bobbyxng and @FabianHofmann was to "just" add the bidding zones as another aggregation level. This is what's added in build_shapes, so that it can be processed by base_network like any other aggregation level. This leads to this edge case of border alignments, that this function solves.

[bobbyxng]

Both approaches are fine with me, basically it's a question of what the smallest "building block" for the BZs is, NUTS3 vs. substation. What's the latest status of your discussion @fneum and @tgilon ?

[tgilon]

I've gone through 3 test cases, which should work in my view. The first two have some issues with the shapes (especially in Italy), which I think are worth revisiting before merging. The third case fails (mixed AC/DC buses). It would be good if @tgilon could trace this error and provide a fix.

Regarding the Italian case, I managed to reproduce a similar issue using a NUTS2 config.

run:
  name: "bidding-zone-4"

scenario:
  clusters:
  - adm
  planning_horizons:
  - 2050

clustering:
  mode: administrative
  administrative:
    level: 2
  copperplate_regions:
  - [DE00, LUG1]
  - [UKNI, IE00]
  build_bidding_zones:
    remove_islands: true
    aggregate_to_tyndp: true
  simplify_network:
    remove_stubs: true
    remove_stubs_across_borders: true

In this case, the same strange shape appears in Italy. This can be explained by examining in the base network that we are using.

Reference using your test config 1

Clustering using NUTS2

Looking at the base network, we can see that there is a line running from Firenze to the south of Volterra. There is also a line running from Prato to the North of Tarquinia. These lines cross. Removing the stubs, we link the Voltera zone with the Arezzo zone (including Firenze).

The same thing happens in southern Italy.

One could think of solving this with remove_stubs_across_borders: false but this would take us back to the AC/DC mix (relates to #1578). Fixing this might may solve the other problems with shapefiles.

[tgilon]

I would recommend setting remove_stubs_across_borders to false by default (and in the test config). This solves our issues.

Here is the associated shapefile (regions_onshore_base_s_adm.geojson):

However, a different approach is required for #1578.

I suggest changing the OSM data. Would it be possible to have a AC bus @bobbyxng on the island ?

This is the raw grid.

Another option would be to force the carrier for this bus using something like this:

n.buses.loc["way/140248154", "carrier"] = "AC"

I'm not a big fan of this one. However, we could hardcode it while we wait for an OSM fix.

[bobbyxng]

I would recommend setting remove_stubs_across_borders to false by default (and in the test config). This solves our issues.

Here is the associated shapefile (regions_onshore_base_s_adm.geojson):

However, a different approach is required for #1578.

I suggest changing the OSM data. Would it be possible to have a AC bus @bobbyxng on the island ?

This is the raw grid.

Another option would be to force the carrier for this bus using something like this:

n.buses.loc["way/140248154", "carrier"] = "AC"

I'm not a big fan of this one. However, we could hardcode it while we wait for an OSM fix.

Hi Thomas! I will look into this. Sorry for the late reply, have been on conference/workshop trips the past two weeks.
Generally it is possible do do this. I checked on openinframap: The reason why the AC bus is missing is because a section between the converter to the 220 kV substation is missing. I can try to fix it on OSM directly, then the scripts should be handle it and we can update the OSM dataset. (Edit: I checked and got this mixed up with Mallorca. Here, Gotland on the AC side only has 110 kV

[tgilon]

Sorry for the late reply, have been on conference/workshop trips the past two weeks.

No problem, it always takes me a while to catch up with new reviews.

I can try to fix it on OSM directly, then the scripts should be handle it and we can update the OSM dataset.

How much time do you need? Should I suggest a quick fix in the meantime (and merge the PR)? Or should we wait for your fix?

[FabianHofmann]

I order to give this a boost (this has already taking quite some resources), I would suggest a quick fix for now with a dependency on the OSM version and try to get the fix into the next version of the OSM.

[bobbyxng]

Regarding my previous comment: I checked and got this mixed up with Mallorca. Gotland on the AC side only has 110 kV. I don't know what the best approach would be here, keeping a 110 kV AC bus would be the easiest and most correct, just need to make sure that it does not cause any issues downstream. I am looking into it now and will propose a solution. Maybe give me until Friday latest so I can make sure everything is fine before pushing a new dataset to Zenodo :) Is there anything else that you want me to focus on in particular? @tgilon @FabianHofmann

[tgilon]

I applied the fix.

This is the final base_network using config/test/config.clusters.yaml.

And this is the associated regions_onshore_base_s_adm.geojson.

Enabling both remove_islands and aggregate_to_tyndp gives this.

[bobbyxng]

@tgilon @FabianHofmann Thanks for the fixes, your proposed temporary hotfix regarding Gotland also works fine. I was just running the entire workflow again and noticed the workflow breaks further downstream in the sector-coupled runs in build_biomass_potentials. Here, data is mapped using the substrings of bus indices, particularly BZ ids "UKXX" clash with "GB" (specifically in the function _calc_unsustainable_potential()). I believe there are a few scripts/rules in the pypsa-eur workflow that rely on this (unfortunately not very robust) approach of extracting substrings from the bus indices.

So to mitigate any of the issues, the easiest fix would be to rename to "UKXX" bidding zones to "GBXX". The alternative would be to adapt all pypsa-eur scripts that rely on this substring mapping.

[bobbyxng]

I added a small condition in build_bidding_zones.py to check whether "IT" is in list of countries. Otherwise the workflow would break if the subset does not contain "IT".

[bobbyxng]

finished reviewing and testing the PR.
Only open points form my side:

• "GB" vs. "UK" substring issue in build_biomass_potentials

[tgilon]

"GB" vs. "UK" substring issue in build_biomass_potentials

Thanks @bobbyxng! I'm currently testing my fix. This has resulted in some other collateral damage.

[tgilon]

@bobbyxng Thank you for the review, suggestion and proactivity! I've fixed the naming convention. This now means that we are no longer following the bidding zone naming convention of the TYNDP 2024. The only exceptions are GB00 and GBNI (rather than UK00 and UKNI).

To ensure everything runs smoothly, I applied same type of fix for the Shetland Islands bus as we did for Öland. Could you please fix this in the next OSM prebuilt?

The workflow runs entirely on my side using config/test/config.clusters.yaml. I had to include the following PR to make it work: #1695.

[fneum]

I'm fine with this version!

I would not have mixed the bidding zone clustering with the administrative shapes, and would have overwritten the aggregated Voronoi regions with the original bidding zone shapefiles after a plain .sjoin operation on the substations.

But this is also functional, and it's generally very useful to have a bidding zone level representation as an option.

• remove_stubs_across_borders should indeed be false.
• I have amended the test config with the renaming of UKNI to GBNI.

Ready to merge?

[tgilon]

Thank you @fneum for your review!

remove_stubs_across_borders should indeed be false.

I left it unspecified so that we could use the default configuration, which is what we need. Completely fine for me to be specific, as this protects the test configuration file against further changes.

I have amended the test config with the renaming of UKNI to GBNI.

Good catch, thanks!

Nothing else to add on my side.

[bobbyxng]

also ready from my side! Thanks @tgilon and @FabianHofmann :)