Major addon rework

[sstroemer]

Alrighty... this is a first draft of the addon rework that we discussed, to be able to talk a bit about the specifics that we may want to change, before everything is set in stone.

Breaking changes for the outside world

• I've bumped to Julia v1.11, anticipating that we may use that in the Python wrapper to load addons why the new "source" entry (which was weird... REPL is now a dependency)
• We are now able to find addons that are: given in a file, already loaded in IESopt, loaded in Main
• Per default, if possible, each addon is reloaded (=> replacing the module), because otherwise users won't see any changes in their addon code reflected.
• One can disable reloading of addons by passing force_reload = false to generate! (or `run!).
• The warning print regarding module replacement is now properly caught and won't show up in iterative solves, even if reloading is active.
• Component-based addons are gone
• Addons (previous global addons) have slightly changed
• "IESopt.component(...)" is now called "IESopt.get_component(...)"; code like component = component(model, str) was just... weird (and error prone)
• Tags are now properly re-introduced (just add "tags: foo" or "tags: [foo, bar]" to any component)
• Tagged components can be extracted by doing IESopt.get_components(model, tagged="foo"); passing ["foo", "bar"] only returns components that have both tags; types (like "Unit" or "CHP") are automatically set
• "Virtual components" that don't exist in the actual model but were "created" by a user (a component with a type that is some template) can now be interacted with; probably not fully featured, but they are first class CoreComponents now

Showcase

Assuming using IESopt was run.

Working with Virtuals

model = generate!(".dev/files/config.iesopt.yaml")
chp = get_components(model, tagged=["CustomCHP"])[1]

chp
# :: Virtual ::
# ├ name: chp
# └ type: CustomCHP

chp.power
# :: Unit ::
# ├ name: chp.power
# ├ inputs: Dict{Carrier, String}(Carrier("gas", nothing, nothing) => "grid_gas")
# ├ outputs: Dict{Carrier, String}(Carrier("electricity", nothing, nothing) => "grid_electricity", Carrier("co2", nothing, nothing) => "grid_co2")
# └ unit_commitment: off

Modifying a Virtual from within a template

Putting this into the finalize part of a template

finalize: |
  access("self").exp.in_gas = access("power").exp.in_gas + access("heat").exp.in_gas

allows doing

chp.exp.in_gas[1]
# 2.5 chp.power.conversion[1] + 0.5 chp.heat.conversion[1]

optimize!(model)
JuMP.value(chp.exp.in_gas[1])
# 9.375

Within an addon

All of that allows also stuff like

chp.con[:CHP_backpressure] = JuMP.@constraint(model, ...)

where chp is the virtual. All of that can after optimizing be accessed.

Things left to do

• remove 18_addon everywhere
• update IESoptLib (chp example, chp template, chp addon)
• update XOR addon to return true (indicating success)
• adapt the Python wrapper for all of this
• this moves to Julia 1.11, which now requires REPL, which may make it inconvenient to support 1.10, which is still in the CI pipeline

[sstroemer]

@daschw (just pinging for after-the-dust-settles, to then go over that together)

[daschw]

Awesome!
Sorry, I just saw this now. I will review this in detail soon, when I find some time.

[sstroemer]

Small update on the internal syntax, after most of it is cleaned up and (at least for the major parts) working:

Templates now properly support dict-valued parameters ...

... which allows configurations like this:

my_component:
  type: InvestableConversionTechnology
  # other stuff ...
  cost:
    invest: 1000.0
    om:
      var: 0.5
      fix:
        rel: 0.05
    lifetime: 25
    rate: 0.07

Getting parameters from within a template ...

... now works as follows (supporting getting/setting and also accessing time series data within the model):

p_min = this.get("p_min", 0)  # automatic support for defaults
p_max = this.get("p_max")

this.set("_cost_om_var_total", cost_om_var)

and since the "wrapper" for these templates (IESopt.Virtual <: IESopt._CoreComponent) works like any other component, dynamically constructing "referrals" between components can make use of the current name of the Virtual

this.set("_ub_single", "100 + $(this.name).invest:value")

Creating first-class math. model items ...

... like expressions, etc., follows the same syntax as everywhere else, allowing the following directly in finalize

this.exp.flow = this.connection_pos.var.flow - this.connection_neg.var.flow

where connection_pos is the directly nested sub-component (not requiring a manual lookup based on its name).

Automatic nested tagging ...

... of Virtuals works as a user might expect it: If Foo is a template that wraps just a single Unit, and Bar wraps Foo to set some (e.g.) sector-specific defaults, then the component

asset:
  type: Bar

will be (internally) flattened all the way down to a Unit. However, tags are created along the way, that allow tracking that asset is actually all of [Unit, Foo, Bar]. Further, if Bar and Foo contain a finalize function that (e.g.) creates convenience results (example: some specialized expressions a user might want to query), they will be called in reverse order: After creating the Unit, Foo.finalize will be called and after that finishes, Bar.finalize can further modify the Virtual.

Note: A warning is in place telling users that this is actually not a good idea, because it may be easy to make mistakes there without fully understanding the internals. So, usage of in-template functions like finalize is recommended to only be used in "actual" templates that contain multiple components.

[sstroemer]

Small note: We could/should use this as an opportunity to cleanup leftover usage of count/size in Decisions.