Sensitivity analysis in ERT

[xjules]

This issue is a work in progress!

Description

Sensitivity analysis is the study of how the uncertainty in the output of a mathematical model or system (numerical or otherwise) can be divided and allocated to different sources of uncertainty in its inputs. This is usually achieved by varying the input parameters and analyzing how much the model response changes.

Current usage

The input usually starts with the design of the analysis case that contains different scenarios introduced in an <design_input.xlsx> file, which is sequentially processed by fmudesign tool to produce an output matrix <output_matrix.xlsl> containing list of parameter realizations.

Forward model DESIGN2PARAMS reads this file and appends the parameter values into <parameters.txt> which is then read by ert.

flowchart LR
    USER((User)) -->|design_input.xlsx| fmudesign(fmudesign)
    ERT -->|parameters.txt| DESIGN2PARAMS
    USER -.->|design_output.xlsx| DESIGN2PARAMS(DESIGN2PARAMS)
    fmudesign -->|design_output.xlsx| DESIGN2PARAMS(DESIGN2PARAMS)
    subgraph fom [Forward Model]
    DESIGN2PARAMS -->|parameters.txt| DESIGN_KW[DESIGN_KW]
    DESIGN2PARAMS -->|parameters.txt| DESIGN_KW1[DESIGN_KW]
    DESIGN2PARAMS -->|parameters.txt| DESIGN_KW2[DESIGN_KW]
    DESIGN2PARAMS -->|parameters.txt| DESIGN_KW3[DESIGN_KW]
    DESIGN_KW -->|global_variables.yaml| RMS
    DESIGN_KW1 -->|RMS_SEED| RMS
    DESIGN_KW2 -->|drogon.multregt| Simulator
    DESIGN_KW3 -->|rate_scaling.yml| RMS
    end

Loading

Step 1: Making ERT capable of reading, storing and outputting sensitivity parameters

flowchart LR
    USER((User)) -->|design_input.xlsx| fmudesign(fmudesign)
    ERT --> parameters[parameters.txt]
    fmudesign -->|design_output.xlsx| ERT
    USER -.->|design_output.xlsx| ERT
    subgraph fom [Forward Model]
    parameters --> DESIGN_KW[DESIGN_KW]
    parameters --> DESIGN_KW1[DESIGN_KW]
    parameters --> DESIGN_KW2[DESIGN_KW]
    parameters --> DESIGN_KW3[DESIGN_KW]
    DESIGN_KW -->|global_variables.yaml| RMS
    DESIGN_KW1 -->|RMS_SEED| RMS
    DESIGN_KW2 -->|drogon.multregt| Simulator
    DESIGN_KW3 -->|rate_scaling.yml| RMS
    end

Loading

Step 2: Sensitivity analysis in ERT

This milestone should alleviate running and using the fmudesign tool by providing a dedicated sensitivity wizard (GUI) in ert capable of specifying custom analysis scenarios yielding parameter specification for ert.

---

Definition of done

Step 1:

• A DESIGN_MATRIX keyword can be used to specify the .xlsx sheet to load sensitivity data from
  - [ ] It should be possible to load .xlsl sheet directly from the GUI in a new run model (starting with ensemble experiment with the possibility to choose "load from")
• All parameters used in the current version of sensitivity analysis is saved in storage and can be plotted
• DESIGN2PARAMS is no longer need when using the DESIGN_MATRIX keyword
• parameters.json and parameters.txt contains the same parameters when using the DESIGN_MATRIX keyword.
• A suggester should pop up that DESIGN_MATRIX can be used when DESIGN2PARAMS is used in config

Step 2:

•  TBD

---

Issues

This should also include validation of the setup such that as many as possible of the following errors should not occur when running the forward models on the compute cluster:

• Validate DESIGN2PARAM configuration at start up instead of when running forward model semeio#638
• Validate DESIGN_KW configuration at start up instead of when running forward model semeio#641
• Implement DESIGN_MATRIX config keyword #8688
• Implement support for categorical data (string) in plotting (ie. use histogram by default) #8687
• Update documentation on DESIGN_MATRIX; the format of the excel sheet especially (see DESIGN2PARAMS also)
• Investigate what happens when DESIGN2PARAMS contains a parameter that is already specified as GEN_KW in the ert config - warning only?
• Update documentation on DESIGN2PARAMS (see semeio)
• Implement GUI option to show all experiment parameters (read from the storage) as a table view

[jonathan-eq]

Just to clarify after some discussion with @sondreso:
The first step of the milestone is to have the xlsx file read and analyse logic from semeio's DESIGN2PARAMS in ert, and instead of writing to parameters.txt, we want to write to ert storage.

[xjules]

Minutes from today's meeting related to mainly step 2 in the current milestone.

• After finishing up DESIGN_MATRIX keyword, we should ideally allow directly to provide design_input.xlsx as the input to DESIGN_MATRIX. This means before starting to work on the UI framework we should get a proper hold on the actual input for fmudesign tooling.
• This would require hardening the current fmu-tools repo by creating more robust tests and general codebase improvements.

[xjules]

More minutes after discussion with @oyvindeide (just unsorted ideas, not necessarily action points) :

• Utilization of DESIGN_MATRIX could be toggled from the GUI

• Integration with GEN_KW is done automatically

  • merging strategy outline:
  • $DM$ = set of DESIGN_MATRIX parameters
  • $KW$ = set of GEN_KW parameters for one parameter group (eg. COEFFS)
  • if $DM\subset KW$ then $DM$ is to be loaded from xlsl and stored as KW parameter group (eg. stored as COEFFS and not as DESIGN_MATRIX)
  • if $DM\bigcap KW\neq \emptyset$ then we raise ValidationError that $DM$ contains some $KW$
  • if $DM\bigcap KW\equiv \emptyset$ then we append $DM$ parameter group to $KW$.

• GEN_KW could be toggled (whether to enable / disable a particular group) from the GUI

• Thus each experiment type should provide running with design matrix (naturally if there are GEN_KWs to update and observations).

• Very rough idea of UI:
  

[xjules]

Minutes from the meeting:
Present: Feda, Jonathan, Sondre, Øyvind, Julius, Lars

The topics were (among others) parameter group names and merging strategies of GEN_KW and DESIGN_MATRIX

• GEN_KW has a template feature, which uses group_name
• We should add logs when and where users use DESIGN2PARAMS and DESIGN_KW
• It would be beneficial to get some example of design_matrices from Bjørn Kåre where he uses multiple instances of DESIGN2PARAMS
• When having both GEN_KW and DESIGN_MATRIX in the ert config:
  • DESIGN_MATRIX has the priority
  • In case GEN_KW = [g:a, g:b, g:c] and DESIGN_MATRIX=[a,b,c] then we should get only a single set of parameters [g:a, g:b, g:c] where the values will come from the DESIGN_MATRIX and the update should be disabled
  • In case GEN_KW = [g:a, g:b, g:c] and DESIGN_MATRIX=[a,b] then we should get only a single set of parameters [g:a, g:b, g:c] where the values [g:a, g:b] will come from the DESIGN_MATRIX and the update should be disabled, while the value of [g:c] will be sampled and update is set to True
  • In case GEN_KW = [g:x, g:y, g:z] and DESIGN_MATRIX=[a,b] then we should get set of parameters [g:a, g:b, g:c] sampled and update set to True and a set of parameters [DESIGN_MATRIX:a, DESIGN_MATRIX:b] which will come from the DESIGN_MATRIX and the update should be disabled.