Feature/optional timestamp (#121)

* Adding chunk index to calculation / estimation results

* Adapted step plots

* Adapted joy plots

* Adapted stacked bar plots

* Added tests

* Update quickstart docs to not use timestamps

* Add tests for runner.py

* Clean up temp dirs after testing

* Move chunk date assignment to inheriting classes (each chunker should decide to support dates or not)

* Display "unified" chunk index (taking reference data into account) in hover

* Updated docs warning about using / not using timestamps in the corresponding code samples

* Updated CHANGELOG.md

* Bump version: 0.6.1 → 0.6.2

* Hopefully fix weird build error due to random rounding??

* Fix linting issue

Co-authored-by: Niels Nuyttens <niels.nuyttens@nannyml.com>

Author: nnansters

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.6.1
+current_version = 0.6.2
 commit = True
 tag = True
 

CHANGELOG.md

@@ -4,6 +4,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.2] - 2022-09-16
+
+### Changed
+
+- Made the `timestamp_column_name` required by all calculators and estimators optional. The main consequences of this
+  are plots have a chunk-index based x-axis now when no timestamp column name was given. You can also not chunk by
+  period when the timestamp column name is not specified.
+
+### Fixed
+
+- Added missing `s3fs` dependency
+- Fixed outdated plotting kind constants in the runner (used by CLI)
+- Fixed some missing images and incorrect version numbers in the README, thanks [@NeoKish](https://github.com/NeoKish)!
+
+### Added
+
+- Added a lot of additional tests, mainly concerning plotting and the [`Runner`](nannyml/runner.py) class
+
 ## [0.6.1] - 2022-09-09
 
 ### Changed

README.md

@@ -69,15 +69,15 @@ Allowing you to have the following benefits:
 | 🔬 **[Technical reference]**                                                                                    | Monitor the performance of your ML models.                                             |
 | 🔎 **[Blog]**                                                                                                   | Thoughts on post-deployment data science from the NannyML team.                        |
 | 📬 **[Newsletter]**                                                                                             | All things post-deployment data science. Subscribe to see the latest papers and blogs. |
-| 💎 **[New in v0.6.1]**                                                                                          | New features, bug fixes.                                                               |
+| 💎 **[New in v0.6.2]**                                                                                          | New features, bug fixes.                                                               |
 | 🧑‍💻 **[Contribute]**                                                                                          | How to contribute to the NannyML project and codebase.                                 |
 | <img src="https://raw.githubusercontent.com/NannyML/nannyml/main/media/slack.png" height='15'> **[Join slack]** | Need help with your specific use case? Say hi on slack!                                |
 
 [NannyML 101]: https://nannyml.readthedocs.io/en/stable/
 [Performance Estimation]: https://nannyml.readthedocs.io/en/stable/how_it_works/performance_estimation.html
 [Key Concepts]: https://nannyml.readthedocs.io/en/stable/glossary.html
 [Technical Reference]:https://nannyml.readthedocs.io/en/stable/nannyml/modules.html
-[New in v0.6.1]: https://github.com/NannyML/nannyml/releases/latest/
+[New in v0.6.2]: https://github.com/NannyML/nannyml/releases/latest/
 [Real World Example]: https://nannyml.readthedocs.io/en/stable/examples/california_housing.html
 [Blog]: https://www.nannyml.com/blog
 [Newsletter]:  https://mailchi.mp/022c62281d13/postdeploymentnewsletter

docs/_static/quick-start-drift-distance_from_office.svg

@@ -391,7 +391,6 @@
     "    y_pred_proba='y_pred_proba',\n",
     "    y_pred='y_pred',\n",
     "    y_true='work_home_actual',\n",
-    "    timestamp_column_name='timestamp',\n",
     "    metrics=['roc_auc'],\n",
     "    chunk_size=chunk_size,\n",
     "    problem_type='classification_binary',\n",
@@ -427,7 +426,6 @@
     "# Let's initialize the object that will perform the Univariate Drift calculations\n",
     "univariate_calculator = nml.UnivariateStatisticalDriftCalculator(\n",
     "    feature_column_names=feature_column_names,\n",
-    "    timestamp_column_name='timestamp',\n",
     "    chunk_size=chunk_size\n",
     ")\n",
     "univariate_calculator = univariate_calculator.fit(reference)\n",
@@ -600,7 +598,6 @@
     "calc = nml.StatisticalOutputDriftCalculator(\n",
     "    y_pred='y_pred',\n",
     "    y_pred_proba='y_pred_proba',\n",
-    "    timestamp_column_name='timestamp',\n",
     "    problem_type='classification_binary'\n",
     ")\n",
     "calc.fit(reference)\n",
@@ -626,7 +623,10 @@
    "outputs": [],
    "source": [
     "# Let's initialize the object that will perform Data Reconstruction with PCA\n",
-    "rcerror_calculator = nml.DataReconstructionDriftCalculator(feature_column_names=feature_column_names, timestamp_column_name='timestamp', chunk_size=chunk_size).fit(reference_data=reference)\n",
+    "rcerror_calculator = nml.DataReconstructionDriftCalculator(\n",
+    "    feature_column_names=feature_column_names,\n",
+    "    chunk_size=chunk_size\n",
+    ").fit(reference_data=reference)\n",
     "# let's see Reconstruction error statistics for all available data\n",
     "rcerror_results = rcerror_calculator.calculate(analysis)\n",
     "figure = rcerror_results.plot(kind='drift', plot_reference=True)\n",

docs/_static/quick-start-drift-gas_price_per_litre.svg

@@ -42,6 +42,12 @@ concepts and functionalities. If you want to know what is implemented under the
 visit :ref:`how it works<how_it_works>`. Finally, if you just look for examples
 on other datasets or ML problems look through our :ref:`examples<examples>`.
 
+.. note::
+    The following example does not use any :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 
 -------------
 Just the code

docs/_static/quick-start-drift-multivariate.svg

@@ -109,6 +109,8 @@ Below we see the columns our dataset contains and explain their purpose.
 +----+------------------------+----------------+-----------------------+------------------------------+--------------------+-----------+----------+
 
 
+.. _data_requirements_columns_timestamp:
+
 Timestamp
 ^^^^^^^^^
 
@@ -124,7 +126,24 @@ In the sample data this is the ``timestamp`` column.
         - *ISO 8601*, e.g. ``2021-10-13T08:47:23Z``
         - *Unix-epoch* in units of seconds, e.g. ``1513393355``
 
-Currently required for all features of NannyML, though we are looking to drop this requirement in a future release.
+
+.. warning::
+    This column is optional. When a timestamp column is not provided, plots will no longer make use of a time based x-axis
+    but will use the index of the chunks instead. The following plots illustrate this:
+
+    .. figure:: /_static/drift-guide-salary_range.svg
+
+        Plot using a time based X-axis
+
+
+    .. figure:: /_static/quick-start-drift-salary_range.svg
+
+        Plot using an index based X-axis
+
+
+    Some :class:`~nannyml.chunk.Chunker` classes might require the presence of a timestamp, such as the
+    :class:`~nannyml.chunk.PeriodBasedChunker`.
+
 
 Target
 ^^^^^^
@@ -183,7 +202,7 @@ You can see those requirements in the table below:
 +--------------+-------------------------------------+-------------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+
 | Data         | Performance Estimation              | Realized Performance                | Univariate Feature Drift          | Multivariate Feature Drift        | Target Drift                      | Output Drift                      |
 +==============+=====================================+=====================================+===================================+===================================+===================================+===================================+
-| timestamp    | Required (reference and analysis)   | Required (reference and analysis)   | Required (reference and analysis) | Required (reference and analysis) | Required (reference and analysis) | Required (reference and analysis) |
+| timestamp    |                                     |                                     |                                   |                                   |                                   |                                   |
 +--------------+-------------------------------------+-------------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+
 | features     |                                     |                                     | Required (reference and analysis) | Required (reference and analysis) |                                   |                                   |
 +--------------+-------------------------------------+-------------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+

docs/_static/quick-start-drift-public_transportation_cost.svg

@@ -13,6 +13,12 @@ If the model's population changes, then its actions will be different.
 The difference in actions is very important to know as soon as possible because
 they directly affect the business results from operating a machine learning model.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 Just The Code
 ------------------------------------
 

docs/_static/quick-start-drift-salary_range.svg

@@ -13,6 +13,12 @@ If the model's population changes, then our populations' actions will be differe
 The difference in actions is very important to know as soon as possible because
 they directly affect the business results from operating a machine learning model.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 
 Just The Code
 ------------------------------------

docs/_static/quick-start-drift-tenure.svg

@@ -13,6 +13,12 @@ If the model's population changes, then the outcome will be different.
 The difference in actions is very important to know as soon as possible because
 they directly affect the business results from operating a machine learning model.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 
 Just The Code
 -------------

docs/_static/quick-start-drift-wfh_prev_workday.svg

@@ -23,6 +23,12 @@ of the available target values for each chunk, for both binary and multiclass cl
 .. note::
     The Target Drift detection process can handle missing target values across all :term:`data periods<Data Period>`.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 
 Just The Code
 ------------------------------------

docs/_static/quick-start-drift-workday.svg

@@ -23,6 +23,12 @@ of the available target values for each chunk, for both binary and multiclass cl
 .. note::
     The Target Drift detection process can handle missing target values across all :term:`data periods<Data Period>`.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 
 Just The Code
 ------------------------------------

docs/_static/quick-start-perf-est.svg

@@ -21,6 +21,12 @@ but also show the target distribution results per chunk with joyploys.
 .. note::
     The Target Drift detection process can handle missing target values across all :term:`data periods<Data Period>`.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 
 Just The Code
 -------------

docs/_static/quick-start-score-drift.svg

@@ -4,6 +4,12 @@
 Monitoring Realized Performance for Binary Classification
 ================================================================
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 Just The Code
 ==============
 

docs/example_notebooks/Quickstart.ipynb

@@ -4,6 +4,12 @@
 Monitoring Realized Performance for Multiclass Classification
 ================================================================
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 
 Just The Code
 ==============

docs/quick.rst

@@ -4,6 +4,12 @@
 Monitoring Realized Performance for Regression
 ==============================================
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 Just The Code
 =============
 

docs/tutorials/data_requirements.rst

@@ -8,6 +8,12 @@ This tutorial explains how to use NannyML to estimate the performance of binary
 models in the absence of target data. To find out how CBPE estimates performance, read the :ref:`explanation of Confidence-based
 Performance Estimation<performance-estimation-deep-dive>`.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 
 .. _performance-estimation-binary-just-the-code:
 

docs/tutorials/detecting_data_drift/model_outputs/drift_detection_for_binary_classification_model_outputs.rst

@@ -8,6 +8,12 @@ This tutorial explains how to use NannyML to estimate the performance of multicl
 models in the absence of target data. To find out how CBPE estimates performance, read the :ref:`explanation of Confidence-based
 Performance Estimation<performance-estimation-deep-dive>`.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 Just The Code
 -------------
 

docs/tutorials/detecting_data_drift/model_outputs/drift_detection_for_multiclass_classification_model_outputs.rst

@@ -8,6 +8,12 @@ This tutorial explains how to use NannyML to estimate the performance of regress
 models in the absence of target data. To find out how DLE estimates performance,
 read the :ref:`explanation of how Direct Loss Estimation works<how-it-works-dle>`.
 
+.. note::
+    The following example uses :term:`timestamps<Timestamp>`.
+    These are optional but have an impact on the way data is chunked and results are plotted.
+    You can read more about them in the :ref:`data requirements<data_requirements_columns_timestamp>`.
+
+
 .. _performance-estimation-regression-just-the-code:
 
 Just The Code

docs/tutorials/detecting_data_drift/model_outputs/drift_detection_for_regression_model_outputs.rst

@@ -32,7 +32,7 @@
 # Dev branch marker is: 'X.Y.dev' or 'X.Y.devN' where N is an integer.
 # 'X.Y.dev0' is the canonical version of 'X.Y.dev'
 #
-__version__ = '0.6.1'
+__version__ = '0.6.2'
 
 import logging
 

docs/tutorials/detecting_data_drift/model_targets/drift_detection_for_binary_classification_model_targets.rst

@@ -66,6 +66,7 @@ def __init__(
         chunk_number: int = None,
         chunk_period: str = None,
         chunker: Chunker = None,
+        timestamp_column_name: Optional[str] = None,
     ):
         """Creates a new instance of an abstract DriftCalculator.
 
@@ -83,7 +84,11 @@ def __init__(
         chunker : Chunker
             The `Chunker` used to split the data sets into a lists of chunks.
         """
-        self.chunker = ChunkerFactory.get_chunker(chunk_size, chunk_number, chunk_period, chunker)
+        self.chunker = ChunkerFactory.get_chunker(
+            chunk_size, chunk_number, chunk_period, chunker, timestamp_column_name
+        )
+
+        self.timestamp_column_name = timestamp_column_name
 
     @property
     def _logger(self) -> logging.Logger:
@@ -167,6 +172,7 @@ def __init__(
         chunk_number: int = None,
         chunk_period: str = None,
         chunker: Chunker = None,
+        timestamp_column_name: str = None,
     ):
         """Creates a new instance of an abstract DriftCalculator.
 
@@ -184,7 +190,10 @@ def __init__(
         chunker : Chunker
             The `Chunker` used to split the data sets into a lists of chunks.
         """
-        self.chunker = ChunkerFactory.get_chunker(chunk_size, chunk_number, chunk_period, chunker)
+        self.chunker = ChunkerFactory.get_chunker(
+            chunk_size, chunk_number, chunk_period, chunker, timestamp_column_name
+        )
+        self.timestamp_column_name = timestamp_column_name
 
     @property
     def _logger(self) -> logging.Logger: