Expose set_config_int, and get_config_int on arcticdb.config, add config docs, documentation fixes

.github/workflows/docs_build.yml

@@ -67,7 +67,7 @@ jobs:
           cd docs/mkdocs
           # stubfiles will be generated into docs/mkdocs/arcticdb_ext and so imported as arcticdb_ext by mkdocs when it builds
           # FIXME --ignore-all-errors may mask new errors and should be removed when we are compliant
-          pybind11-stubgen arcticdb_ext.version_store --ignore-all-errors -o .
+          pybind11-stubgen arcticdb_ext --ignore-all-errors -o .
 
       - name: List docs versions before deploy
         run: |

cpp/arcticdb/python/python_module.cpp

@@ -142,9 +142,19 @@ void register_log(py::module && log) {
 void register_configs_map_api(py::module& m) {
     using namespace arcticdb;
 #define EXPOSE_TYPE(LABEL, TYPE) \
-    m.def("get_config_" #LABEL, [](const std::string& label) { return ConfigsMap::instance()->get_##LABEL(label); }); \
-    m.def("set_config_" #LABEL, [](const std::string& label, TYPE value)  { ConfigsMap::instance()->set_##LABEL(label, value); }); \
-    m.def("unset_config_" #LABEL, [](const std::string& label)  { ConfigsMap::instance()->unset_##LABEL(label); });
+    m.def("get_config_" #LABEL, \
+        [](const std::string& label) { return ConfigsMap::instance()->get_##LABEL(label); }, \
+        "Get configured value, returns None if not set.", \
+        py::arg("label")); \
+    m.def("set_config_" #LABEL, \
+        [](const std::string& label, TYPE value)  { ConfigsMap::instance()->set_##LABEL(label, value); }, \
+        "Set configured value.", \
+        py::arg("label"), \
+        py::arg("value")); \
+    m.def("unset_config_" #LABEL, \
+        [](const std::string& label)  { ConfigsMap::instance()->unset_##LABEL(label); }, \
+        "Unset configured value.", \
+        py::arg("label"));
 
     EXPOSE_TYPE(int, int64_t)
     EXPOSE_TYPE(string, std::string)

docs/README.md

@@ -17,17 +17,24 @@ and diagrams are generated, even undocumented classes / functions. This gives ab
 
 ### mkdocs
 
-Install
-```
-pip3 install mkdocs-material mkdocs-jupyter 'mkdocstrings[python]' black pybind11-stubgen mike wheel
-```
+Dependencies
+
 - mkdocs-material: theme
 - mkdocs-jupyter: jupyter notebook support
 - mkdocstrings[python]: python docstring support, like sphinx docs
 - black: for signature formatting
 - pybind11-stubgen: for generating stubs for pybind11 extensions, so that mkdocstrings can parse them
 - mike: for deploying versioned docs
 
+Install
+```
+pip3 install mkdocs-material mkdocs-jupyter 'mkdocstrings[python]' black pybind11-stubgen mike wheel
+```
+or
+```
+mamba install mkdocs-material mkdocs-jupyter mkdocstrings-python black pybind11-stubgen mike wheel
+```
+
 You need to have ArcticDB installed to generate the API docs, so install it from source:
 
 ```
@@ -45,18 +52,18 @@ pip install arcticdb
 ```
 cd docs/mkdocs
 # TODO fix these errors!
-pybind11-stubgen arcticdb_ext.version_store --ignore-all-errors -o .
+pybind11-stubgen arcticdb_ext --ignore-all-errors -o .
 ```
 
 To build the latest mkdocs to docs/mkdocs/site:
 ```
 cd docs/mkdocs
-mkdocs build -s
+mkdocs build
 ```
 
 Development server 
 ```
-mkdocs serve -s -a 0.0.0.0:8000
+mkdocs serve -a 0.0.0.0:8000
 ```
 
 The python docstring parser, griffe, will struggle with some built-in (C++) modules

docs/mkdocs/docs/api/config.md

@@ -0,0 +1,7 @@
+# Config API
+
+::: arcticdb.config.get_config_int
+
+::: arcticdb.config.set_config_int
+
+::: arcticdb.config.set_log_level

docs/mkdocs/docs/aws.md

@@ -55,12 +55,15 @@ arctic = adb.Arctic('s3://s3.<REGION>.amazonaws.com:<BUCKET_NAME>?aws_auth=true'
 
 ## Checking connectivity to the S3 bucket.
 
-ArcticDB currently uses five S3 methods, those are:
-- GetObject
-- PutObject
-- HeadObject
-- DeleteObject
-- ListObjectsV2
+ArcticDB currently uses five S3 methods.
+
+ Method       | Permission required
+------------- | -------------------
+GetObject     | s3:GetObject
+HeadObject    | s3:GetObject
+PutObject     | s3:PutObject
+DeleteObject  | s3:DeleteObject
+ListObjectsV2 | s3:ListBucket
 
 Here is a short script you can use to check connectivity to a bucket from your client.  If this works, then the configuration should be correct for read and write with ArcticDB.
 

docs/mkdocs/docs/faq.md

@@ -130,7 +130,7 @@ Pickled data cannot be index or column-sliced, and neither `update` nor `append`
 
 ### *How does indexing work in ArcticDB?*
 
-See the [Getting Started](index.md#reading-and-writing-dataframes) page for details of supported index types.
+See the [Getting Started](index.md#reading-and-writing-data) page for details of supported index types.
 
 ### *Can I `append` with additional columns / What is Dynamic Schema?*
 
@@ -140,7 +140,7 @@ You can also change the type of numerical columns - for example, integers will b
 
 ### *How does ArcticDB segment data?*
 
-See [On Disk Storage Format](technical/on_disk_storage.md) and the [documentation](api/arctic.md#LibraryOptions) for the `rows_per_segment` and `columns_per_segment` library configuration options for more details. 
+See [On Disk Storage Format](technical/on_disk_storage.md) and the [documentation](api/arctic.md#arcticdb.LibraryOptions) for the `rows_per_segment` and `columns_per_segment` library configuration options for more details. 
 
 ### *How does ArcticDB handle streaming data?*
 
@@ -182,4 +182,4 @@ The handling of `NaN` in ArcticDB depends on the type of the column under consid
 
 * For string columns, `NaN`, as well as Python `None`, are fully supported.
 * For floating-point numeric columns, `NaN` is also fully supported.
-* For integer numeric columns `NaN` is not supported. A column that otherwise contains only integers will be treated as a floating point column if a `NaN` is encountered by ArcticDB, at which point [the usual rules](api/arctic.md#LibraryOptions) around type promotion for libraries configured with or without dynamic schema all apply as usual.
+* For integer numeric columns `NaN` is not supported. A column that otherwise contains only integers will be treated as a floating point column if a `NaN` is encountered by ArcticDB, at which point [the usual rules](api/arctic.md#arcticdb.LibraryOptions) around type promotion for libraries configured with or without dynamic schema all apply as usual.

docs/mkdocs/docs/index.md

@@ -96,7 +96,7 @@ library = ac.get_library('intro', create_if_missing=True)
 
     The one exception where schemas are needed is in the case of functions that modify existing symbols: `update` and `append`. When modifying a symbol, the new data must have the same schema as the existing data. The schema here means the index type and the name, order, and type of each column in the DataFrame. In other words when you are appending new rows they must look like the existing rows. This is the default option and is called `static schema`.
 
-    However, if you need to add, remove or change the type of columns via `update` or `append`, then you can do that. You simply need to create the library with the `dynamic_schema` option set. See the `library_options` parameter of the ([`create_library`](api/arctic/#arcticdb.Arctic.create_library)) method.
+    However, if you need to add, remove or change the type of columns via `update` or `append`, then you can do that. You simply need to create the library with the `dynamic_schema` option set. See the `library_options` parameter of the ([`create_library`](api/arctic.md#arcticdb.Arctic.create_library)) method.
 
     So you have the best of both worlds - you can choose to either enforce a static schema on your data so it cannot be changed by modifying operations, or allow it to be flexible.
 
@@ -388,7 +388,7 @@ Note the timestamps - we've read the data prior to the `append` operation. Pleas
 
     This behaviour can be can be controlled via the `prune_previous_versions` keyword argument. Space will be saved but the previous versions will then not be available.
 
-    A compromise can be achieved by using snapshots, which allow states of the library to be saved and read back later. This allows certain versions to be protected from deletion, they will be deleted when the snapshot is deleted. See [snapshot documentation](api/library/#arcticdb.version_store.library.Library.snapshot) for details.
+    A compromise can be achieved by using snapshots, which allow states of the library to be saved and read back later. This allows certain versions to be protected from deletion, they will be deleted when the snapshot is deleted. See [snapshot documentation](api/library.md#arcticdb.version_store.library.Library.snapshot) for details.
 
 
 

docs/mkdocs/docs/notebooks/ArcticDB_billion_row_challenge.ipynb

@@ -38,11 +38,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from arcticdb_ext import set_config_int\n",
+    "from arcticdb.config import set_config_int\n",
     "set_config_int('VersionStore.NumCPUThreads', 16)"
    ]
   },

docs/mkdocs/docs/runtime_config.md

@@ -11,7 +11,7 @@ All of the integer options detailed on this page can be configured using the fol
 For integer options, the following code snippet demonstrates how to set values in code:
 
 ```python
-from arcticdb_ext import set_config_int
+from arcticdb.config import set_config_int
 set_config_int(setting, value)
 ```
 

docs/mkdocs/docs/technical/upgrade_storage.md

@@ -25,9 +25,11 @@ anything, but will log the affected libraries with "Config NOT OK for <library_n
 
 If no libraries were shown as affected after following the steps above, you can stop now. You do not need to do any more.
 
-> :warning: Running this script will break access for clients on less than version 3.0.0 of ArcticDB for affected libraries.
-> The affected libraries were shown in the step above.
-> Ensure users have upgraded to at least version `arcticdb==3.0.0` first.
+!!! warning
+
+    Running this script will break access for clients on less than version 3.0.0 of ArcticDB for affected libraries.
+    The affected libraries were shown in the step above.
+    Ensure users have upgraded to at least version `arcticdb==3.0.0` first.
 
 - Run `arcticdb_update_storage --uri "<uri>" --run` where `<uri>` is the same as the one above.
 

docs/mkdocs/mkdocs.yml

@@ -4,6 +4,14 @@ repo_url: https://github.com/man-group/ArcticDB
 repo_name: man-group/ArcticDB
 remote_branch: docs-pages
 
+strict: true
+
+validation:
+  omitted_files: warn
+  absolute_links: warn
+  unrecognized_links: warn
+  anchors: warn
+
 markdown_extensions:
   - admonition
   - pymdownx.details
@@ -103,6 +111,7 @@ nav:
     - On-Disk Storage: 'technical/on_disk_storage.md'
     - Arctic Migration: 'technical/migration.md'
     - Contributing: 'technical/contributing.md'
+    - Upgrade Storage: 'technical/upgrade_storage.md'
   - Configuration:
     - Library Configuration: 'lib_config.md'
     - Runtime Configuration: 'runtime_config.md'
@@ -125,3 +134,9 @@ nav:
   - Library related objects: 'api/library_types.md'
   - DataFrame Processing Operations API: 'api/processing.md'
   - Exceptions: 'api/exceptions.md'
+  - Config: 'api/config.md'
+
+exclude_docs: |
+  *.py
+  /notebooks/README.md
+  technical/linting.md

python/arcticdb/config.py

@@ -25,6 +25,7 @@
 
 from arcticdb.exceptions import ArcticNativeException
 from arcticdb.log import logger_by_name, configure
+from arcticdb_ext import set_config_int, get_config_int
 
 _HOME = osp.expanduser("~/.arctic/native")
 
@@ -161,7 +162,7 @@ def save_loggers_config(config=None, path=Defaults.LOG_CONF_FILE_PATH):
 
 
 def make_loggers_config(
-    default_level=Defaults.DEFAULT_LOG_LEVEL,
+    default_level: str = Defaults.DEFAULT_LOG_LEVEL,
     specific_log_levels: Optional[Dict[str, str]] = None,
     console_output: bool = True,
     file_output_path: Optional[str] = None,
@@ -171,14 +172,15 @@ def make_loggers_config(
 
     Parameters
     ----------
-    default_level
+    default_level: str
         Default log level for all the loggers unless overriden with specific_log_levels.
         Valid values are "DEBUG", "INFO", "WARN", "ERROR".
-    specific_log_levels
-        Optional overrides for specific logger(s). The possible logger names can be found in log.py.
-    console_output
+    specific_log_levels: Optional[Dict[str, str]]
+        Optional overrides for specific logger(s).
+        The possible logger names can be found in `arcticdb.log.logger_by_name.keys()` (subject to change).
+    console_output: bool
         Boolean indicating whether to output logs to the terminal.
-    file_output_path
+    file_output_path: str
         If None, logs will not be written to a file. Otherwise, this value should be set to the path of a file to which
         logging output will be written.
 
@@ -216,12 +218,27 @@ def make_loggers_config(
 
 
 def set_log_level(
-    default_level=Defaults.DEFAULT_LOG_LEVEL, specific_log_levels=None, console_output=True, file_output_path=None
+    default_level: str = Defaults.DEFAULT_LOG_LEVEL,
+    specific_log_levels: Optional[Dict[str, str]] = None,
+    console_output: bool = True,
+    file_output_path: Optional[str] = None,
 ):
     """
-    Passes the arguments to ``make_loggers_config`` and then configures the loggers, overwriting any existing config.
+    Set log levels, overwriting any existing config.
 
-    For more information on the parameters this method takes, please see the documentation for `make_loggers_config`.
+    Parameters
+    ----------
+    default_level: str
+        Default log level for all the loggers unless overriden with specific_log_levels.
+        Valid values are "DEBUG", "INFO", "WARN", "ERROR".
+    specific_log_levels: Optional[Dict[str, str]]
+        Optional overrides for specific logger(s).
+        The possible logger names can be found in `arcticdb.log.logger_by_name.keys()` (subject to change).
+    console_output: bool
+        Boolean indicating whether to output logs to the terminal.
+    file_output_path: str
+        If None, logs will not be written to a file. Otherwise, this value should be set to the path of a file to which
+        logging output will be written.
     """
     return configure(
         make_loggers_config(default_level, specific_log_levels, console_output, file_output_path), force=True

python/arcticdb/version_store/library.py

@@ -1191,7 +1191,7 @@ def update_batch(
     ) -> List[Union[VersionedItem, DataError]]:
         """
         Perform an update operation on a list of symbols in parallel. All constrains on
-        [update](/api/library/#arcticdb.version_store.library.Library.update) apply to this call as well.
+        [update](#arcticdb.version_store.library.Library.update) apply to this call as well.
 
         Parameters
         ----------
@@ -1578,9 +1578,6 @@ def read(
             Defer query execution until `collect` is called on the returned `LazyDataFrame` object. See documentation
             on `LazyDataFrame` for more details.
 
-        output_format: OutputFormat, default=OutputFormat.PANDAS:
-            What format to return the output in. One of PANDAS or ARROW.
-
         Returns
         -------
         Union[VersionedItem, LazyDataFrame]