Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major addon rework #20

Draft
wants to merge 40 commits into
base: main
Choose a base branch
from
Draft

Conversation

sstroemer
Copy link
Member

@sstroemer sstroemer commented Oct 11, 2024

Alrighty... this is a first draft of the addon rework that we discussed, to be able to talk a bit about the specifics that we may want to change, before everything is set in stone.

Breaking changes for the outside world

  • I've bumped to Julia v1.11, anticipating that we may use that in the Python wrapper to load addons why the new "source" entry (which was weird... REPL is now a dependency)
  • We are now able to find addons that are: given in a file, already loaded in IESopt, loaded in Main
  • Per default, if possible, each addon is reloaded (=> replacing the module), because otherwise users won't see any changes in their addon code reflected.
  • One can disable reloading of addons by passing force_reload = false to generate! (or `run!).
  • The warning print regarding module replacement is now properly caught and won't show up in iterative solves, even if reloading is active.
  • Component-based addons are gone
  • Addons (previous global addons) have slightly changed
  • "IESopt.component(...)" is now called "IESopt.get_component(...)"; code like component = component(model, str) was just... weird (and error prone)
  • Tags are now properly re-introduced (just add "tags: foo" or "tags: [foo, bar]" to any component)
  • Tagged components can be extracted by doing IESopt.get_components(model, tagged="foo"); passing ["foo", "bar"] only returns components that have both tags; types (like "Unit" or "CHP") are automatically set
  • "Virtual components" that don't exist in the actual model but were "created" by a user (a component with a type that is some template) can now be interacted with; probably not fully featured, but they are first class CoreComponents now

Showcase

Assuming using IESopt was run.

Working with Virtuals

model = generate!(".dev/files/config.iesopt.yaml")
chp = get_components(model, tagged=["CustomCHP"])[1]

chp
# :: Virtual ::
# ├ name: chp
# └ type: CustomCHP

chp.power
# :: Unit ::
# ├ name: chp.power
# ├ inputs: Dict{Carrier, String}(Carrier("gas", nothing, nothing) => "grid_gas")
# ├ outputs: Dict{Carrier, String}(Carrier("electricity", nothing, nothing) => "grid_electricity", Carrier("co2", nothing, nothing) => "grid_co2")
# └ unit_commitment: off

Modifying a Virtual from within a template

Putting this into the finalize part of a template

finalize: |
  access("self").exp.in_gas = access("power").exp.in_gas + access("heat").exp.in_gas

allows doing

chp.exp.in_gas[1]
# 2.5 chp.power.conversion[1] + 0.5 chp.heat.conversion[1]

optimize!(model)
JuMP.value(chp.exp.in_gas[1])
# 9.375

Within an addon

All of that allows also stuff like

chp.con[:CHP_backpressure] = JuMP.@constraint(model, ...)

where chp is the virtual. All of that can after optimizing be accessed.

Things left to do

  • remove 18_addon everywhere
  • update IESoptLib (chp example, chp template, chp addon)
  • update XOR addon to return true (indicating success)
  • adapt the Python wrapper for all of this
  • this moves to Julia 1.11, which now requires REPL, which may make it inconvenient to support 1.10, which is still in the CI pipeline

@sstroemer
Copy link
Member Author

@daschw (just pinging for after-the-dust-settles, to then go over that together)

@daschw
Copy link
Collaborator

daschw commented Oct 16, 2024

Awesome!
Sorry, I just saw this now. I will review this in detail soon, when I find some time.

@sstroemer
Copy link
Member Author

Small update on the internal syntax, after most of it is cleaned up and (at least for the major parts) working:

Templates now properly support dict-valued parameters ...

... which allows configurations like this:

my_component:
  type: InvestableConversionTechnology
  # other stuff ...
  cost:
    invest: 1000.0
    om:
      var: 0.5
      fix:
        rel: 0.05
    lifetime: 25
    rate: 0.07

Getting parameters from within a template ...

... now works as follows (supporting getting/setting and also accessing time series data within the model):

p_min = this.get("p_min", 0)  # automatic support for defaults
p_max = this.get("p_max")

this.set("_cost_om_var_total", cost_om_var)

and since the "wrapper" for these templates (IESopt.Virtual <: IESopt._CoreComponent) works like any other component, dynamically constructing "referrals" between components can make use of the current name of the Virtual

this.set("_ub_single", "100 + $(this.name).invest:value")

Creating first-class math. model items ...

... like expressions, etc., follows the same syntax as everywhere else, allowing the following directly in finalize

this.exp.flow = this.connection_pos.var.flow - this.connection_neg.var.flow

where connection_pos is the directly nested sub-component (not requiring a manual lookup based on its name).

Automatic nested tagging ...

... of Virtuals works as a user might expect it: If Foo is a template that wraps just a single Unit, and Bar wraps Foo to set some (e.g.) sector-specific defaults, then the component

asset:
  type: Bar

will be (internally) flattened all the way down to a Unit. However, tags are created along the way, that allow tracking that asset is actually all of [Unit, Foo, Bar]. Further, if Bar and Foo contain a finalize function that (e.g.) creates convenience results (example: some specialized expressions a user might want to query), they will be called in reverse order: After creating the Unit, Foo.finalize will be called and after that finishes, Bar.finalize can further modify the Virtual.

Note: A warning is in place telling users that this is actually not a good idea, because it may be easy to make mistakes there without fully understanding the internals. So, usage of in-template functions like finalize is recommended to only be used in "actual" templates that contain multiple components.

sstroemer added a commit that referenced this pull request Oct 23, 2024
…es in 1.11 (will be reverted as soon as #20 is merged)
sstroemer and others added 26 commits October 23, 2024 20:11
addons now properly check if they are already loaded (force_reload is now a general argument to all external model creation functions)

addons are found in IESopt, but also as modules manually loaded into Main
…ing of a CoreComponent

this is a workaround, and should be reverted as soon as possible, since REPL is not an actually necessary dependency
…model components" that are templates (and no real component)
…for Virtuals for now)

small fix in how addons are called before the full refactoring
…to in during `finalize`, by calling `access("self")`
Stefan Strömer and others added 2 commits October 25, 2024 18:03
@sstroemer
Copy link
Member Author

Small note: We could/should use this as an opportunity to cleanup leftover usage of count/size in Decisions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants