Merge pull request #2 from pepkit/dev

v0.0.3

.github/workflows/run-pytest.yml

@@ -10,7 +10,7 @@ jobs:
   pytest:
     strategy:
       matrix:
-        python-version: [3.6, 3.7, 3.8]
+        python-version: [3.6, 3.7, 3.8, 3.9]
         os: [ubuntu-latest] # can't use macOS when using service containers or container jobs
     runs-on: ${{ matrix.os }}
     services:

README.md

@@ -1,5 +1,6 @@
 ![Run pytests](https://github.com/pepkit/pipestat/workflows/Run%20pytests/badge.svg)
 [![codecov](https://codecov.io/gh/pepkit/pipestat/branch/master/graph/badge.svg?token=O07MXSQZ32)](https://codecov.io/gh/pepkit/pipestat)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
 <img src="https://raw.githubusercontent.com/pepkit/pipestat/master/docs/img/pipestat_logo.svg?sanitize=true" alt="pipestat" height="70"/><br>
 

codecov.yml

@@ -0,0 +1,5 @@
+ignore:
+  - "*/argparser.py"
+  - "*/cli.py"
+  - "*/__main__.py"
+  - "setup.py"

docs/README.md

@@ -1,5 +1,6 @@
 ![Run pytests](https://github.com/pepkit/pipestat/workflows/Run%20pytests/badge.svg)
 [![codecov](https://codecov.io/gh/pepkit/pipestat/branch/master/graph/badge.svg?token=O07MXSQZ32)](https://codecov.io/gh/pepkit/pipestat)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
 <img src="https://raw.githubusercontent.com/pepkit/pipestat/master/docs/img/pipestat_logo.svg?sanitize=true" alt="pipestat" height="70"/><br>
 

docs/api_docs.md

@@ -461,4 +461,4 @@ Check schema for any possible issues
 
 
 
-*Version Information: `pipestat` v0.0.1, generated by `lucidoc` v0.4.3*
+*Version Information: `pipestat` v0.0.3-dev, generated by `lucidoc` v0.4.3*

docs/autodoc_build/pipestat.md

@@ -339,7 +339,7 @@ Result schema mappings
 
 
 ```python
-def retrieve(self, record_identifier=None, result_identifier=None, limit=None)
+def retrieve(self, record_identifier=None, result_identifier=None)
 ```
 
 Retrieve a result for a record.
@@ -350,7 +350,6 @@ be returned.
 
 - `record_identifier` (`str`):  unique identifier of the record
 - `result_identifier` (`str`):  name of the result to be retrieved
-- `limit` (`int`):  max number of results to be returned
 
 
 #### Returns:
@@ -385,7 +384,7 @@ Schema path
 
 
 ```python
-def select(self, columns=None, condition=None, condition_val=None, limit=None)
+def select(self, columns=None, condition=None, condition_val=None, offset=None, limit=None)
 ```
 
 Get all the contents from the selected table, possibly restricted by the provided condition.
@@ -394,7 +393,8 @@ Get all the contents from the selected table, possibly restricted by the provide
 - `columns` (`str | list[str]`):  columns to select
 - `condition` (`str`):  condition to restrict the resultswith, will be appended to the end of the SELECT statement and safely populated with 'condition_val', for example: `"id=%s"`
 - `condition_val` (`list`):  values to fill the placeholderin 'condition' with
-- `limit` (`int`):  max number of results to be returned
+- `offset` (`int`):  number of records to be skipped
+- `limit` (`int`):  max number of records to be returned
 
 
 #### Returns:
@@ -460,4 +460,4 @@ Check schema for any possible issues
 
 
 
-*Version Information: `pipestat` v0.0.1, generated by `lucidoc` v0.4.3*
+*Version Information: `pipestat` v0.0.3-dev, generated by `lucidoc` v0.4.3*

docs/changelog.md

@@ -2,6 +2,11 @@
 
 This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format. 
 
+## [0.0.3] - 2021-03-12
+### Added
+- possibility to initialize the `PipestatManager` object (or use the `pipestat status` CLI) with no results schema defined for pipeline status management; [Issue #1](https://github.com/pepkit/pipestat/issues/1) 
+
+
 ## [0.0.2] - 2021-02-22
 ### Added
 - initial package release

docs_jupyter/cli.ipynb

@@ -51,7 +51,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "version: 0.0.1\n",
+      "version: 0.0.3\n",
       "usage: pipestat [-h] [--version] [--silent] [--verbosity V] [--logdev]\n",
       "                {report,inspect,remove,retrieve,status} ...\n",
       "\n",
@@ -518,8 +518,8 @@
       "\n",
       "\n",
       "PipestatManager (test)\n",
-      "Backend: file (/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmp.XSWvUccu)\n",
-      "Results schema source: /Users/mstolarczyk/code/pipestat/docs_jupyter/../tests/data/sample_output_schema.yaml\n",
+      "Backend: file (/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmp.Zid7BMd1)\n",
+      "Results schema source: ../tests/data/sample_output_schema.yaml\n",
       "Status schema source: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pipestat/schemas/status_schema.yaml\n",
       "Records count: 1\n"
      ]
@@ -548,8 +548,8 @@
       "\n",
       "\n",
       "PipestatManager (test)\n",
-      "Backend: file (/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmp.XSWvUccu)\n",
-      "Results schema source: /Users/mstolarczyk/code/pipestat/docs_jupyter/../tests/data/sample_output_schema.yaml\n",
+      "Backend: file (/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmp.Zid7BMd1)\n",
+      "Results schema source: ../tests/data/sample_output_schema.yaml\n",
       "Status schema source: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pipestat/schemas/status_schema.yaml\n",
       "Records count: 1\n",
       "\n",
@@ -655,8 +655,8 @@
       "\n",
       "\n",
       "PipestatManager (test)\n",
-      "Backend: file (/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmp.XSWvUccu)\n",
-      "Results schema source: /Users/mstolarczyk/code/pipestat/docs_jupyter/../tests/data/sample_output_schema.yaml\n",
+      "Backend: file (/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmp.Zid7BMd1)\n",
+      "Results schema source: ../tests/data/sample_output_schema.yaml\n",
       "Status schema source: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pipestat/schemas/status_schema.yaml\n",
       "Records count: 1\n",
       "\n",
@@ -680,7 +680,9 @@
     "To manage pipeline status call `pipestat status <subcommand>`:\n",
     "\n",
     "- `set` to set pipeline statuses\n",
-    "- `get` to retrieve pipeline statuses"
+    "- `get` to retrieve pipeline statuses\n",
+    "\n",
+    "Starting with `pipestat 0.0.3` the `--schema` argument is not required for status management if YAML file is used as the backend."
    ]
   },
   {

docs_jupyter/python_api.ipynb

@@ -57,12 +57,13 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmp_fb7s787.yaml\n"
+      "/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmpa6poai5_.yaml\n"
      ]
     }
    ],
    "source": [
     "from tempfile import mkstemp\n",
+    "\n",
     "_, temp_file = mkstemp(suffix=\".yaml\")\n",
     "print(temp_file)"
    ]
@@ -81,10 +82,10 @@
    "outputs": [],
    "source": [
     "psm = pipestat.PipestatManager(\n",
-    "    namespace=\"test\", \n",
-    "    record_identifier=\"sample1\", \n",
-    "    results_file_path=temp_file, \n",
-    "    schema_path=\"../tests/data/sample_output_schema.yaml\"\n",
+    "    namespace=\"test\",\n",
+    "    record_identifier=\"sample1\",\n",
+    "    results_file_path=temp_file,\n",
+    "    schema_path=\"../tests/data/sample_output_schema.yaml\",\n",
     ")"
    ]
   },
@@ -283,7 +284,7 @@
     }
    ],
    "source": [
-    "try: \n",
+    "try:\n",
     "    psm.report(values={\"output_file\": {\"path\": \"/home/user/path.csv\"}})\n",
     "except ValidationError as e:\n",
     "    print(e)"
@@ -322,7 +323,12 @@
    ],
    "source": [
     "psm.report(\n",
-    "  values={\"output_file\": {\"path\": \"/home/user/path.csv\", \"title\": \"CSV file with some data\"}}\n",
+    "    values={\n",
+    "        \"output_file\": {\n",
+    "            \"path\": \"/home/user/path.csv\",\n",
+    "            \"title\": \"CSV file with some data\",\n",
+    "        }\n",
+    "    }\n",
     ")"
    ]
   },
@@ -389,7 +395,12 @@
    ],
    "source": [
     "psm.report(\n",
-    "  values={\"output_file\": {\"path\": \"/home/user/path_new.csv\", \"title\": \"new CSV file with some data\"}}\n",
+    "    values={\n",
+    "        \"output_file\": {\n",
+    "            \"path\": \"/home/user/path_new.csv\",\n",
+    "            \"title\": \"new CSV file with some data\",\n",
+    "        }\n",
+    "    }\n",
     ")"
    ]
   },
@@ -425,8 +436,13 @@
    ],
    "source": [
     "psm.report(\n",
-    "  values={\"output_file\": {\"path\": \"/home/user/path_new.csv\", \"title\": \"new CSV file with some data\"}},\n",
-    "  force_overwrite=True\n",
+    "    values={\n",
+    "        \"output_file\": {\n",
+    "            \"path\": \"/home/user/path_new.csv\",\n",
+    "            \"title\": \"new CSV file with some data\",\n",
+    "        }\n",
+    "    },\n",
+    "    force_overwrite=True,\n",
     ")\n",
     "psm.data"
    ]
@@ -445,10 +461,10 @@
    "outputs": [],
    "source": [
     "psm1 = pipestat.PipestatManager(\n",
-    "  namespace=\"test\",\n",
-    "  record_identifier=\"sample1\",\n",
-    "  results_file_path=temp_file,\n",
-    "  schema_path=\"../tests/data/sample_output_schema.yaml\"\n",
+    "    namespace=\"test\",\n",
+    "    record_identifier=\"sample1\",\n",
+    "    results_file_path=temp_file,\n",
+    "    schema_path=\"../tests/data/sample_output_schema.yaml\",\n",
     ")"
    ]
   },
@@ -492,7 +508,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmp_fb7s787.yaml\n",
+      "/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmpa6poai5_.yaml\n",
       "test:\n",
       "  sample1:\n",
       "    output_file:\n",
@@ -590,9 +606,7 @@
    "source": [
     "try:\n",
     "    psm.report(\n",
-    "      record_identifier=\"sample2\",\n",
-    "      values={\"number_of_things\": []},\n",
-    "      strict_type=False\n",
+    "        record_identifier=\"sample2\", values={\"number_of_things\": []}, strict_type=False\n",
     "    )\n",
     "except TypeError as e:\n",
     "    print(e)"
@@ -814,20 +828,21 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmpmny6226c.yaml\n"
+      "/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmpf9m81e1n.yaml\n"
      ]
     }
    ],
    "source": [
     "from tempfile import mkstemp\n",
+    "\n",
     "_, temp_file_highlight = mkstemp(suffix=\".yaml\")\n",
     "print(temp_file_highlight)\n",
     "\n",
     "psm_highlight = pipestat.PipestatManager(\n",
-    "    namespace=\"test_highlight\", \n",
-    "    record_identifier=\"sample1\", \n",
-    "    results_file_path=temp_file_highlight, \n",
-    "    schema_path=\"../tests/data/sample_output_schema_highlight.yaml\"\n",
+    "    namespace=\"test_highlight\",\n",
+    "    record_identifier=\"sample1\",\n",
+    "    results_file_path=temp_file_highlight,\n",
+    "    schema_path=\"../tests/data/sample_output_schema_highlight.yaml\",\n",
     ")"
    ]
   },
@@ -999,6 +1014,96 @@
     "\n",
     "Please refer to the Python API documentation (`__init__` method) to see how to use custom status schema and flags directory."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Initializing `PipestatManager` without results schema\n",
+    "\n",
+    "Starting with `pipestat 0.0.3`, it is possible to initialize the `PipestatManager` object without specifying the results schema file. This feature comes in handy if `PipestatManager` is created with a sole intent to monitor pipeline status.\n",
+    "\n",
+    "Here's an example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/tmpq81inuvn.yaml\n"
+     ]
+    }
+   ],
+   "source": [
+    "_, temp_file_no_schema = mkstemp(suffix=\".yaml\")\n",
+    "print(temp_file_no_schema)\n",
+    "\n",
+    "psm_no_schema = pipestat.PipestatManager(\n",
+    "    namespace=\"test_no_schema\", results_file_path=temp_file_no_schema\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As you can see, the object has been initialized successfully. Obviuosly, the schema has to be defined to report and retrieve results as the requirement to predefine results and therefore the possibility to rely on the schema to gather all the possible results metadata in the pipestat clients is a big advantage.\n",
+    "\n",
+    "Moreover, initialization with a database as a backend is impossible without schema due to the characteristics of relational databases, which must have columns predefined, when created."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 33,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Results schema not found. The schema is required to report results. It needs to be supplied to the object constructor.\n"
+     ]
+    }
+   ],
+   "source": [
+    "try:\n",
+    "    psm_no_schema.report(record_identifier=\"sample1\", values={\"key\": \"val\"})\n",
+    "except pipestat.SchemaNotFoundError as e:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As mentioned above, the pipeline status management capabilities are supported with no results schema defined:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'running'"
+      ]
+     },
+     "execution_count": 34,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "psm_no_schema.set_status(status_identifier=\"running\", record_identifier=\"sample1\")\n",
+    "psm_no_schema.get_status(record_identifier=\"sample1\")"
+   ]
   }
  ],
  "metadata": {

pipestat/__init__.py

@@ -1,9 +1,10 @@
 # Project configuration, particularly for logging.
 
 import logmuse
+
 from ._version import __version__
-from .pipestat import *
 from .helpers import *
+from .pipestat import *
 
 __classes__ = ["PipestatManager"]
 __all__ = __classes__