Merge latest from develop (#51)

* Fix scale factor for Unit set to Fraction

* Attempt to fix issue with containment plots failing to show up initially

* Update encrypted contact information

* Improve hover-information

* Fix bug that removed legend text in containment plots

* Fix bug with UNSMRY assuming an older csv format

* Add ContainmentDataProvider wrapper class

Intended to wrap an EnsembleTableProvider that reads containment
summary files (either csv or arrow)

* Add UnsmryDataProvider wrapper class

Intended to wrap an EnsembleTableProvider that reads unsmry
summary files (either csv or arrow)

* Add validation to containment classes

* Change summary file read order

Arrow format is attempted first instead of csv

* Change summary file read order

Fix issue with unsmry option and missing files

* Fix issue with legend when first real is not "0"

* Fix bug with unsmry reset

* Remove large csv file check

* Remove unused file size function

* Changes to adapt CO2Leakage for residual trapping maps visualization

* Format changes

* Solve minor issues

* Solve typo

* Fix issue in property_origin

* Minor fixes

* Rename MapGroup CO2_MASS

* Add visualization menu with separate filters for each attribute

* Formatting with mypy, pylint and black

* Simplify MapThresholds class

* Fix visualization threshold bug for plume maps

* Add mean in tornado table VolumetricAnalysis (equinor#1297)

* Fix merge conflict after rebase

* Fix issues related to missing polygon data

* Fix black format issues

* Fix another conflict from rebase

* Fix another rebase conflict

* Handle missing well picks and mypy errors

* Remove surface cache file mode (equinor#1298)

* OS temp path for image server (equinor#1302)

* Add missing file handling for well picks

* Enable usage without either maps or containment tables

* CCS-194: Fix after change of units for co2 mass. (#34)

* Tweak visualization filter

* Add polygon provider

Basically a copy of the provider for fault polygons. Renamed
"fault polygon" to "polygon" and added a couple of TODOs

* CCS-213: Handle plume_groups (#36)

* CCS-213: Start using plume group from containment results.

* CCS-213: Cont using plume group from containment results.

* CCS-213: Fixes for plume groups.

* CCS-213: Add plume groups to filter.

* CCS-213: Some minor fixes.

* CCS-213: Minor fix.

* Some tests.

* CCS-213: Join lines before/after merge when coloring by plume groups.

* CCS-213: Some refactoring.

* CCS-213: Fix combine plume group with none as mark.

* CCS-213: Minor fix.

* CCS-213: Remove some prints.

* CCS-213: Use starting 0-values for unmerged wells.

* CCS-213: Fix for multiple realization.

* CCS-213: Fix case without plume groups.

* CCS-213: Minor fix.

* CCS-213: Fix black.

* CCS-213: Some refactoring.

* CCS-213: Minor change, remove temp code.

* Combine time plots, add mean/P10/P90 option

* Fix formatting

* Fix missing issues with EnsemblePolygonProvider

* Distinguish p10 and p90 from mean, turn select-all-realizations into button

* Introduce new polygon provider in plugin

* Fix black/lint/mypy errorss

* CCS-213: Rename "?" to "undetermined" and fix warning messages (#38)

* CCS-213: Fix warning message.

* CCS-213: Rename ? to undetermined.

* Fix issues found while testing

* Minor fix in _get_menu_options() (#39)

* Minor fix in _get_menu_options().

* New minor fix.

* Minor change. (#40)

* CCS-212: Renaming (#41)

* CCS-212: Renaming, graph plots.

* CCS-212: Renaming, map plots.

* CCS-212: More renaming, map plots.

* CCS-212: More renaming, map plots.

* CCS-191: Add statistics plot (#44)

* CCS-191: Start on statistics plot.

* CCS-191: Minor change.

* CCS-191: Use color and mark options.

* CCS-191: Cont use color and mark options.

* CCS-191: Cont using color and mark options.

* CCS-191: Fix in label names.

* CCS-191: Add default category plotted, single option.

* CCS-191: Minor fig fixes.

* CCS-191: Add hovering.

* CCS-191: Fix some filtering.

* CCS-191: Remove prints and comments.

* CCS-191: Fill date options, use in plot.

* CCS-191: Same fixes for date, and use in first plot.

* CCS-191: Change some names, add titles.

* CCS-191: Clean up.

* CCS-191: Fix in set_date_option().

* Black

* isort

* CCS-191: Minor change.

* CCS-212: Renaming, changing same labels, changing back gas_phase to sgas etc, add code for XMF2 (#45)

* CCS-212: Use old file names for sgas etc maps.

* CCS-212: Allow XMF2-files as default in addition to AMFG. Some changes to labels for 2D maps.

* CCS-212: Some clean up.

* CCS-212: Add SGSTR as option.

* CCS-212: Remove prints.

* Fix linting (#49)

* Fix linting

* Fix linting 2

* Fix linting 3

* Fix linting 4

* Fix linting 5

* Fix linting 6

* Fix linting 7

* Fix linting 8

* Develop master merged (#50)

* Add mean in tornado table VolumetricAnalysis (equinor#1297)

* Remove surface cache file mode (equinor#1298)

* OS temp path for image server (equinor#1302)

* Update README.md with deprecation warning (equinor#1303)

* No Active Rfts error message (equinor#1304)

Co-authored-by: Øyvind Lind-Johansen <olind@equinor.com>

* Pandas 2 compatibility for RFT plotter (equinor#1305)

* Pandas 2 compability for RFT plotter

* black

* Remove use of `wcc.ColorScales` (equinor#1306)

* Fix FutureWarning - cast to dtype (equinor#1309)

* Pandas 2 compability for ParameterResponseCorrelation (equinor#1310)

* Add field outline and custom well picks colors to MapViewerFMU (equinor#1311)

* Replaced use of `DashSubsurfaceViewer` with `SubsurfaceViewer` (equinor#1312)

* Expose rft input files as arguments (equinor#1313)

* exposed rft input files as arguments

* black version updated

* pylint fixes

* pylint fixes

* more fixes

---------

Co-authored-by: Øyvind Lind-Johansen <olind@equinor.com>

* Disable cache for statistical surfaces (equinor#1314)

* Add check for length of numpy array (equinor#1316)

* Fix ensemble colors in relperm (equinor#1317)

---------

Co-authored-by: FredrikNevjenNR <fnevjen@nr.no>
Co-authored-by: FredrikNevjenNR <150343101+FredrikNevjenNR@users.noreply.github.com>
Co-authored-by: Vegard Kvernelv <vegard@nr.no>
Co-authored-by: Jorge Sicacha <sicacha@nr.no>
Co-authored-by: jorgesicachanr <114475076+jorgesicachanr@users.noreply.github.com>
Co-authored-by: vegardkv <vkvernelv@gmail.com>

webviz_subsurface/_providers/ensemble_polygon_provider/__init__.py

@@ -0,0 +1,3 @@
+from .ensemble_polygon_provider import EnsemblePolygonProvider, SimulatedPolygonsAddress
+from .ensemble_polygon_provider_factory import EnsemblePolygonProviderFactory
+from .polygon_server import PolygonsAddress, PolygonServer

webviz_subsurface/_providers/ensemble_polygon_provider/_polygon_discovery.py

@@ -0,0 +1,97 @@
+import glob
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Dict, List, Optional
+
+# The fmu.ensemble dependency resdata is only available for Linux,
+# hence, ignore any import exception here to make
+# it still possible to use the PvtPlugin on
+# machines with other OSes.
+#
+# NOTE: Functions in this file cannot be used
+#       on non-Linux OSes.
+try:
+    from fmu.ensemble import ScratchEnsemble
+except ImportError:
+    pass
+
+
+@dataclass(frozen=True)
+class PolygonsFileInfo:
+    path: str
+    real: int
+    name: str
+    attribute: str
+
+
+def _discover_ensemble_realizations_fmu(ens_path: str) -> Dict[int, str]:
+    """Returns dict indexed by realization number and with runpath as value"""
+    scratch_ensemble = ScratchEnsemble("dummyEnsembleName", paths=ens_path).filter("OK")
+    real_dict = {i: r.runpath() for i, r in scratch_ensemble.realizations.items()}
+    return real_dict
+
+
+def _discover_ensemble_realizations(ens_path: str) -> Dict[int, str]:
+    # Much faster than FMU impl above, but is it risky?
+    # Do we need to check for OK-file?
+    real_dict: Dict[int, str] = {}
+
+    realidxregexp = re.compile(r"realization-(\d+)")
+    globbed_real_dirs = sorted(glob.glob(str(ens_path)))
+    for real_dir in globbed_real_dirs:
+        realnum: Optional[int] = None
+        for path_comp in reversed(real_dir.split(os.path.sep)):
+            realmatch = re.match(realidxregexp, path_comp)
+            if realmatch:
+                realnum = int(realmatch.group(1))
+                break
+
+        if realnum is not None:
+            real_dict[realnum] = real_dir
+
+    return real_dict
+
+
+@dataclass(frozen=True)
+class PolygonsIdent:
+    name: str
+    attribute: str
+
+
+def _polygons_ident_from_filename(filename: str) -> Optional[PolygonsIdent]:
+    """Split the stem part of the fault polygons filename into fault polygons name and attribute"""
+    delimiter: str = "--"
+    parts = Path(filename).stem.split(delimiter)
+    if len(parts) != 2:
+        return None
+
+    return PolygonsIdent(name=parts[0], attribute=parts[1])
+
+
+def discover_per_realization_polygons_files(
+    ens_path: str,
+    polygons_pattern: str,
+) -> List[PolygonsFileInfo]:
+    polygons_files: List[PolygonsFileInfo] = []
+
+    real_dict = _discover_ensemble_realizations_fmu(ens_path)
+    for realnum, runpath in sorted(real_dict.items()):
+        if Path(polygons_pattern).is_absolute():
+            filenames = [polygons_pattern]
+        else:
+            filenames = glob.glob(str(Path(runpath) / polygons_pattern))
+        for polygons_filename in sorted(filenames):
+            polygons_ident = _polygons_ident_from_filename(polygons_filename)
+            if polygons_ident:
+                polygons_files.append(
+                    PolygonsFileInfo(
+                        path=polygons_filename,
+                        real=realnum,
+                        name=polygons_ident.name,
+                        attribute=polygons_ident.attribute,
+                    )
+                )
+
+    return polygons_files

webviz_subsurface/_providers/ensemble_polygon_provider/_provider_impl_file.py

@@ -0,0 +1,226 @@
+import logging
+import shutil
+from pathlib import Path
+from typing import List, Optional
+
+import pandas as pd
+import xtgeo
+
+from webviz_subsurface._utils.enum_shim import StrEnum
+from webviz_subsurface._utils.perf_timer import PerfTimer
+
+from ._polygon_discovery import PolygonsFileInfo
+from .ensemble_polygon_provider import (
+    EnsemblePolygonProvider,
+    PolygonsAddress,
+    SimulatedPolygonsAddress,
+)
+
+LOGGER = logging.getLogger(__name__)
+
+REL_SIM_DIR = "sim"
+
+
+# pylint: disable=too-few-public-methods
+class Col:
+    TYPE = "type"
+    REAL = "real"
+    ATTRIBUTE = "attribute"
+    NAME = "name"
+    ORIGINAL_PATH = "original_path"
+    REL_PATH = "rel_path"
+
+
+class PolygonType(StrEnum):
+    SIMULATED = "simulated"
+    HAZARDUOUS_BOUNDARY = "hazarduous_boundary"
+    CONTAINMENT_BOUNDARY = "containment_boundary"
+
+
+class ProviderImplFile(EnsemblePolygonProvider):
+    def __init__(
+        self,
+        provider_id: str,
+        provider_dir: Path,
+        polygon_inventory_df: pd.DataFrame,
+    ) -> None:
+        self._provider_id = provider_id
+        self._provider_dir = provider_dir
+        self._inventory_df = polygon_inventory_df
+
+    @staticmethod
+    # pylint: disable=too-many-locals
+    def write_backing_store(
+        storage_dir: Path,
+        storage_key: str,
+        sim_polygons: List[PolygonsFileInfo],
+    ) -> None:
+        timer = PerfTimer()
+
+        # All data for this provider will be stored inside a sub-directory
+        # given by the storage key
+        provider_dir = storage_dir / storage_key
+        LOGGER.debug(f"Writing polygon backing store to: {provider_dir}")
+        provider_dir.mkdir(parents=True, exist_ok=True)
+        (provider_dir / REL_SIM_DIR).mkdir(parents=True, exist_ok=True)
+
+        type_arr: List[PolygonType] = []
+        real_arr: List[int] = []
+        attribute_arr: List[str] = []
+        name_arr: List[str] = []
+        rel_path_arr: List[str] = []
+        original_path_arr: List[str] = []
+
+        for polygon_info in sim_polygons:
+            rel_path_in_store = _compose_rel_sim_polygons_path(
+                real=polygon_info.real,
+                attribute=polygon_info.attribute,
+                name=polygon_info.name,
+                extension=Path(polygon_info.path).suffix,
+            )
+            type_arr.append(PolygonType.SIMULATED)
+            real_arr.append(polygon_info.real)
+            attribute_arr.append(polygon_info.attribute)
+            name_arr.append(polygon_info.name)
+            rel_path_arr.append(str(rel_path_in_store))
+            original_path_arr.append(polygon_info.path)
+
+        LOGGER.debug(f"Copying {len(original_path_arr)} polygons into backing store...")
+        timer.lap_s()
+        _copy_polygons_into_provider_dir(original_path_arr, rel_path_arr, provider_dir)
+        et_copy_s = timer.lap_s()
+
+        polygons_inventory_df = pd.DataFrame(
+            {
+                Col.TYPE: type_arr,
+                Col.REAL: real_arr,
+                Col.ATTRIBUTE: attribute_arr,
+                Col.NAME: name_arr,
+                Col.REL_PATH: rel_path_arr,
+                Col.ORIGINAL_PATH: original_path_arr,
+            }
+        )
+
+        parquet_file_name = provider_dir / "polygons_inventory.parquet"
+        polygons_inventory_df.to_parquet(path=parquet_file_name)
+
+        LOGGER.debug(
+            f"Wrote polygon backing store in: {timer.elapsed_s():.2f}s ("
+            f"copy={et_copy_s:.2f}s)"
+        )
+
+    @staticmethod
+    def from_backing_store(
+        storage_dir: Path,
+        storage_key: str,
+    ) -> Optional["ProviderImplFile"]:
+        provider_dir = storage_dir / storage_key
+        parquet_file_name = provider_dir / "polygons_inventory.parquet"
+
+        try:
+            polygons_inventory_df = pd.read_parquet(path=parquet_file_name)
+            return ProviderImplFile(storage_key, provider_dir, polygons_inventory_df)
+        except FileNotFoundError:
+            return None
+
+    def provider_id(self) -> str:
+        return self._provider_id
+
+    def attributes(self) -> List[str]:
+        return sorted(list(self._inventory_df[Col.ATTRIBUTE].unique()))
+
+    def fault_polygons_names_for_attribute(self, polygons_attribute: str) -> List[str]:
+        return sorted(
+            list(
+                self._inventory_df.loc[
+                    self._inventory_df[Col.ATTRIBUTE] == polygons_attribute
+                ][Col.NAME].unique()
+            )
+        )
+
+    def realizations(self) -> List[int]:
+        unique_reals = self._inventory_df[Col.REAL].unique()
+
+        # Sort and strip out any entries with real == -1
+        return sorted([r for r in unique_reals if r >= 0])
+
+    def get_polygons(
+        self,
+        address: PolygonsAddress,
+    ) -> Optional[xtgeo.Polygons]:
+        if isinstance(address, SimulatedPolygonsAddress):
+            return self._get_simulated_polygons(address)
+
+        raise TypeError("Unknown type of fault polygons address")
+
+    def _get_simulated_polygons(
+        self, address: SimulatedPolygonsAddress
+    ) -> Optional[xtgeo.Polygons]:
+        """Returns a Xtgeo fault polygons instance of a single realization fault polygons"""
+
+        timer = PerfTimer()
+
+        polygons_fns: List[Path] = self._locate_simulated_polygons(
+            attribute=address.attribute,
+            name=address.name,
+            realizations=[address.realization],
+        )
+
+        if len(polygons_fns) == 0:
+            LOGGER.warning(f"No simulated polygons found for {address}")
+            return None
+        if len(polygons_fns) > 1:
+            LOGGER.warning(
+                f"Multiple simulated polygonss found for: {address}"
+                "Returning first fault polygons."
+            )
+
+        if polygons_fns[0].suffix == ".csv":
+            polygons = xtgeo.Polygons(pd.read_csv(polygons_fns[0]))
+        else:
+            polygons = xtgeo.polygons_from_file(polygons_fns[0])
+
+        LOGGER.debug(f"Loaded simulated fault polygons in: {timer.elapsed_s():.2f}s")
+
+        return polygons
+
+    def _locate_simulated_polygons(
+        self, attribute: str, name: str, realizations: List[int]
+    ) -> List[Path]:
+        """Returns list of file names matching the specified filter criteria"""
+        df = self._inventory_df.loc[
+            self._inventory_df[Col.TYPE] == PolygonType.SIMULATED
+        ]
+
+        df = df.loc[
+            (df[Col.ATTRIBUTE] == attribute)
+            & (df[Col.NAME] == name)
+            & (df[Col.REAL].isin(realizations))
+        ]
+
+        return [self._provider_dir / rel_path for rel_path in df[Col.REL_PATH]]
+
+
+def _copy_polygons_into_provider_dir(
+    original_path_arr: List[str],
+    rel_path_arr: List[str],
+    provider_dir: Path,
+) -> None:
+    for src_path, dst_rel_path in zip(original_path_arr, rel_path_arr):
+        # LOGGER.debug(f"copying fault polygons from: {src_path}")
+        shutil.copyfile(src_path, provider_dir / dst_rel_path)
+
+    # full_dst_path_arr = [storage_dir / dst_rel_path for dst_rel_path in store_path_arr]
+    # with ProcessPoolExecutor() as executor:
+    #     executor.map(shutil.copyfile, original_path_arr, full_dst_path_arr)
+
+
+def _compose_rel_sim_polygons_path(
+    real: int,
+    attribute: str,
+    name: str,
+    extension: str,
+) -> Path:
+    """Compose path to simulated fault polygons file, relative to provider's directory"""
+    fname = f"{real}--{name}--{attribute}{extension}"
+    return Path(REL_SIM_DIR) / fname

webviz_subsurface/_providers/ensemble_polygon_provider/ensemble_polygon_provider.py

@@ -0,0 +1,53 @@
+import abc
+from dataclasses import dataclass
+from typing import List, Optional
+
+import xtgeo
+
+
+@dataclass(frozen=True)
+class SimulatedPolygonsAddress:
+    """Specifies a unique simulated polygon set for a given ensemble realization"""
+
+    attribute: str
+    name: str
+    realization: int
+
+
+# Type aliases used for signature readability
+PolygonsAddress = SimulatedPolygonsAddress
+
+
+# Class provides data for ensemble surfaces
+class EnsemblePolygonProvider(abc.ABC):
+    @abc.abstractmethod
+    def provider_id(self) -> str:
+        """Returns string ID of the provider."""
+
+    @abc.abstractmethod
+    def attributes(self) -> List[str]:
+        """Returns list of all available attributes."""
+
+    @abc.abstractmethod
+    def realizations(self) -> List[int]:
+        """Returns list of all available realizations."""
+
+    @abc.abstractmethod
+    def get_polygons(
+        self,
+        address: PolygonsAddress,
+    ) -> Optional[xtgeo.Polygons]:
+        """Returns fault polygons for a given fault polygons address"""
+
+    # @abc.abstractmethod
+    # def get_surface_bounds(self, surface: EnsembleSurfaceContext) -> List[float]:
+    #     """Returns the bounds for a surface [xmin,ymin, xmax,ymax]"""
+
+    # @abc.abstractmethod
+    # def get_surface_value_range(self, surface: EnsembleSurfaceContext) -> List[float]:
+    #     """Returns the value range for a given surface context [zmin, zmax]"""
+
+    # @abc.abstractmethod
+    # def get_surface_as_rgba(self, surface: EnsembleSurfaceContext) -> io.BytesIO:
+    #     """Returns surface as a greyscale png RGBA with encoded elevation values
+    #     in a bytestream"""