Added option to specify GEN_OBS nodes in the form nodename:index

This makes it possible to specify individual observations from an GEN_OBS node
in localisation.
Add tests to verify active param and active obs in ministeps

semeio/workflows/localisation/local_config_script.py

@@ -1,26 +1,25 @@
-from ert_shared.libres_facade import LibresFacade
 from ert_shared.plugins.plugin_manager import hook_implementation
 
 import semeio.workflows.localisation.local_script_lib as local
 from semeio.communication import SemeioScript
-from semeio.workflows.localisation.localisation_config import LocalisationConfig
+from semeio.workflows.localisation.localisation_config import (
+    LocalisationConfig,
+    get_max_gen_obs_size_for_expansion,
+)
 
 
 class LocalisationConfigJob(SemeioScript):
     def run(self, *args):
         ert = self.ert()
-        facade = LibresFacade(self.ert())
+
         # Clear all correlations
         local.clear_correlations(ert)
 
         # Read yml file with specifications
         config_dict = local.read_localisation_config(args)
 
-        # Get all observations from ert instance
-        obs_keys = [
-            facade.get_observation_key(nr)
-            for nr, _ in enumerate(facade.get_observations())
-        ]
+        expand_gen_obs_max_size = get_max_gen_obs_size_for_expansion(config_dict)
+        obs_keys = local.get_obs_from_ert(ert, expand_gen_obs_max_size)
 
         ert_parameters = local.get_param_from_ert(ert.ensembleConfig())
 
@@ -35,8 +34,17 @@ def run(self, *args):
             ert_parameters.to_dict(),
             ert.getLocalConfig(),
             ert.ensembleConfig(),
+            ert.getObservations(),
             ert.eclConfig().getGrid(),
         )
+        if config.verify_active:
+            local.verify_ministep_active_param(
+                config.correlations,
+                ert.getLocalConfig(),
+                ert.ensembleConfig(),
+                ert_parameters.to_dict(),
+            )
+            local.verify_ministep_active_obs(config.correlations, ert)
 
 
 DESCRIPTION = """
@@ -56,8 +64,9 @@ def run(self, *args):
 Features
 ----------
 The following features are implemented:
-
- - The user defines groups of model parameters and observations.
+ - The user defines groups of model parameters and observations,
+   called correlation groups or ministeps. It is possible to specify many correlation
+   groups.
  - Wildcard notation can be used to specify a selection of model parameter groups
    and observation groups.
  - For scalar parameters coming from the ERT keywords GEN_KW and GEN_PARAM,
@@ -68,8 +77,9 @@ def run(self, *args):
    value corresponding to a grid cell (i,j,k) in location (x,y,z) is reduced by a
    scaling factor varying by distance from a reference point e.g at a location (X,Y,Z),
    usually specified to be close to an observation group.
- - Multiple pairs of groups of model parameters and observations can be specified
-   to have active correlations.
+ - A requirement is that a pair of observation and model parameter (obs, param)
+   is only appearing once to avoid double specification of the same active
+   correlation.
 
 
 Using the localisation setup in ERT
@@ -84,27 +94,35 @@ def run(self, *args):
  - Specify to automatically run the workflow after the initial ensemble is created,
    but before the first update by using the HOOK_WORKFLOW keyword
    with the option PRE_FIRST_UPDATE.
+ - To QC the specification of the config file for localisation, it is possible to
+   run the workflow before running initial ensemble also, but due to limitations
+   in ERT implementation GEN_PARAM type of parameter nodes will have empty
+   list of parameters if the workflow is run before initialization. If  GEN_PARAM
+   nodes are used in correlation groups, an error message may appear.
+
 """
 
 EXAMPLES = """
-Example configuration
+Example configurations
 -------------------------
 
 The configuration file is a YAML format file where pairs of groups of observations
 and groups of model parameters are specified.
 
-Per default, all correlations between the
-observations from the observation group and model parameters from the model
-parameter group are active and unmodified. All other combinations of pairs of
-observations and model parameters not specified in a correlation group, are inactive
-or set to 0. But it is possible to specify many correlation groups. If a pair of
-observation and model parameter appear multiple times
+Per default, all correlations between the observations from the observation 
+group and model parameters from the model parameter group are active 
+and unmodified. All other combinations of pairs of observations and model 
+parameters not specified in a correlation group, are inactive or set to 0. 
+But it is possible to specify many correlation groups. If a pair of observation 
+and model parameter appear multiple times
 (e.g. because they are member of multiple correlation groups),
 an error message is raised.
 
 It is also possible to scale down correlations that are specified for 3D and 2D fields.
 
-In the example below, four correlation groups are defined.
+Example 1:
+------------
+In the first example below, four correlation groups are defined.
 The first correlation group is called ``CORR1`` (a user defined name),
 and defines all observations to have active correlation with all model
 parameters starting with ``aps_valysar_grf`` and with ``GEO:PARAM``.
@@ -170,6 +188,50 @@ def run(self, *args):
           active_segments: [ 1,2,4]
           scalingfactors: [1.0, 0.5, 0.3]
 
+Example 2:
+------------
+In this example the optional keyword **max_gen_obs_size** is specified. 
+The value 1000 means that all observation nodes of type GEN_OBS having less 
+than 1000 observations are specified in the form::
+
+ nodename:index 
+
+where **index** is an integer from 0 to 999. 
+All GEN_OBS nodes with more than 1000 observations
+are specified in the form nodename only. The reason not to enable to specify
+individual observations from GEN_OBS of any size is performance e.g. when 
+GEN_OBS nodes of seismic data is used. The first example below (2A) 
+specifies all observations by:: 
+
+ GENOBS_NODE:*
+
+The second example (2B) has selected a few observations from the 
+GENOBS_NODE::
+
+  ["GENOBS_NODE:0","GENOBS_NODE:3","GENOBS_NODE:55"]
+
+Example 2A::
+
+  max_gen_obs_size: 1000
+  log_level:2
+  correlations:
+    - name: CORR1
+       obs_group:
+          add: ["GENOBS_NODE:*"]
+       param_group:
+          add: ["PARAM_NODE:*"]
+
+Example 2B::
+
+  max_gen_obs_size: 100
+  log_level:2
+  correlations:
+    - name: CORR1
+       obs_group:
+          add: ["GENOBS_NODE:0","GENOBS_NODE:3","GENOBS_NODE:55"]
+       param_group:
+          add: ["PARAM_NODE:*"]
+
 
 Keywords
 -----------
@@ -187,6 +249,16 @@ def run(self, *args):
       and make it possible to visualise them. Is only relevant when using
       **field_scale** with methods calculating the scaling factors.
 
+:max_gen_obs_size:
+      Specify the max size of GEN_OBS type of observation nodes that
+      can specify individual observations. Individual observations are specified
+      by nodename:index where index is the observation number in the 
+      observation file associated with the GEN_OBS type node.
+      The keyword is optional. If not specified or specified with value 0,
+      this means that observations of type GEN_OBS is specified by
+      nodename only. Individual observations can not be specified in this case
+      which means that all observations in the GEN_OBS node is used.
+
 :correlations:
       List of specifications of correlation groups. A correlation group
       specify a set of observations and a set of model parameters.