New commands for CLI: sample-config, report-config, init, results ; Refactoring CLI

docs/reference/cli.md

@@ -1,6 +1,16 @@
 # NePS Command Line Interface
 This section provides a brief overview of the commands available in the NePS CLI.
 
+!!! note "Support of Development and Task ID"
+    The NePS arguments `development_stage_id` and `task_id` are only partially
+    supported. To retrieve results for a specific task or development stage, you must modify the `root_directory` to
+    point to the corresponding folder of your NePS results. For example, if you have task_id 1 and development_stage_id 4,
+    update your root_directory to root_directory/task_1/development_4. This can be done either by specifying the
+    --root-directory option in your command or by updating the root_directory in your corresponding `run_args` yaml
+    file.
+
+
+---
 ## **`init` Command**
 
 Generates a default `run_args` YAML configuration file, providing a template that you can customize for your experiments.
@@ -11,7 +21,7 @@ Generates a default `run_args` YAML configuration file, providing a template tha
 - `-h, --help` (Optional): show this help message and exit
 - `--config-path` (Optional): Optional custom path for generating the configuration file. Default is 'run_config.yaml'.
 - `--template` (Optional): Optional, options between different templates. Required configs(basic) vs all neps configs (complete)
-- `--state-machine` (Optional): If set, creates a NEPS state. Requires an existing config.yaml.
+- `--database` (Optional):  If set, creates the NePS database. This is required if you want to sample and report configurations using only CLI commands. Requires an existing config.yaml.
 
 
 **Example Usage:**
@@ -20,7 +30,7 @@ Generates a default `run_args` YAML configuration file, providing a template tha
 neps init --config-path custom/path/config.yaml --template complete
 ```
 
-
+---
 ## **`run` Command**
 
 Executes the optimization based on the provided configuration. This command serves as a CLI wrapper around `neps.run`, effectively mapping each CLI argument to a parameter in `neps.run`. It offers a flexible interface that allows you to override the existing settings specified in the YAML configuration file, facilitating dynamic adjustments for managing your experiments.
@@ -56,15 +66,15 @@ Executes the optimization based on the provided configuration. This command serv
 neps run --run-args path/to/config.yaml --max-evaluations-total 50
 ```
 
-
+---
 ## **`status` Command**
 Check the status of the NePS run. This command provides a summary of trials, including pending, evaluating, succeeded, and failed trials. You can filter the trials displayed based on their state.
 
 **Arguments:**
 
 
 - `-h, --help` (Optional): show this help message and exit
-- `--root-directory` (Optional): Optional: The path to your root_directory. If not provided, it will be loaded from run_config.yaml.
+- `--root-directory` (Optional): The path to your root_directory. If not provided, it will be loaded from run_config.yaml.
 - `--pending` (Optional): Show only pending trials.
 - `--evaluating` (Optional): Show only evaluating trials.
 - `--succeeded` (Optional): Show only succeeded trials.
@@ -75,7 +85,7 @@ Check the status of the NePS run. This command provides a summary of trials, inc
 neps status --root-directory path/to/directory --succeeded
 ```
 
-
+---
 ## **`info-config` Command**
 Provides detailed information about a specific configuration identified by its ID. This includes metadata, configuration values, and trial status.
 
@@ -94,16 +104,24 @@ Provides detailed information about a specific configuration identified by its I
 neps info-config 42 --root-directory path/to/directory
 ```
 
+---
 ## **`results` Command**
 Displays the results of the NePS run, listing all incumbent trials in reverse order (most recent first). Optionally,
-you can plot the results to visualize the progression of incumbents over trials.
+you can plot the results to visualize the progression of incumbents over trials.  Additionally, you can dump all
+trials or incumbent trials to a file in the specified format and plot the results to visualize the progression of
+incumbents over trials.
 
 **Arguments:**
 
 
 - `-h, --help` (Optional): show this help message and exit
 - `--root-directory` (Optional): Optional: The path to your root_directory. If not provided, it will be loaded from run_config.yaml.
-- `--plot` (Optional): Plot the results if set.
+- `--plot` (Optional): Plot the incumbents if set.
+- `--dump-all-configs` (Optional): Dump all information about the trials to a file in the specified format (csv, json,
+  parquet).
+- `--dump-incumbents` (Optional): Dump only the information about the incumbent trials to a file in the specified
+  format (csv, json, parquet).
+
 
 
 **Example Usage:**
@@ -112,8 +130,7 @@ you can plot the results to visualize the progression of incumbents over trials.
 neps results --root-directory path/to/directory --plot
 ```
 
-
-
+---
 ## **`errors` Command**
 Lists all errors found in the specified NePS run. This is useful for debugging or reviewing failed trials.
 
@@ -131,23 +148,61 @@ neps errors --root-directory path/to/directory
 ```
 
 
+---
 ## **`sample-config` Command**
+The sample-config command allows users to generate new configurations based on the current state of the
+NePS optimizer. This is particularly useful when you need to manually intervene in the sampling process, such
+as allocating different computational resources to different configurations.
 
+!!! note "Note"
+    Before using the `sample-config` command, you need to initialize the database by running `neps init --database` if you haven't already executed `neps run`. Running `neps run` will also create a `NePsState`.
 
 **Arguments:**
 
-
 - `-h, --help` (Optional): show this help message and exit
-- `--root-directory` (Optional): Optional: The path to your root_directory. If not provided, it will be loaded from run_config.yaml.
+- `--worker-id` (Optional): The worker ID for which the configuration is being sampled.
+- `--run-args` (Optional): Path to the YAML configuration file. If not provided, it will search after run_config.yaml.
+- `--number-of-configs` (Optional): Number of configurations to sample (default: 1).
 
 
 **Example Usage:**
 
+
 ```bash
-neps sample-config --help
+neps sample-config --worker-id worker_1 --number-of-configs 5
 ```
 
+---
+## **`report-config` Command**
+The `report-config` command is the counterpart to `sample-config` and reports the outcome of a specific trial by updating its status and associated metrics in the NePS state. This command is crucial for manually managing the evaluation results of sampled configurations.
+
+**Arguments:**
+
+
+- `` (Required): ID of the trial to report
+- `` (Required): Outcome of the trial
+
+
+- `-h, --help` (Optional): show this help message and exit
+- `--worker-id` (Optional): The worker ID for which the configuration is being sampled.
+- `--loss` (Optional): Loss value of the trial
+- `--run-args` (Optional): Path to the YAML file containing run configurations
+- `--cost` (Optional): Cost value of the trial
+- `--learning-curve` (Optional): Learning curve as a list of floats, provided like this --learning-curve 0.9 0.3 0.1
+- `--duration` (Optional): Duration of the evaluation in sec
+- `--err` (Optional): Error message if any
+- `--tb` (Optional): Traceback information if any
+- `--time-end` (Optional): The time the trial ended as either a UNIX timestamp (float) or in 'YYYY-MM-DD HH:MM:SS' format
+
+
+**Example Usage:**
+
+
+```bash
+neps report-config 42 success --worker-id worker_1 --loss 0.95 --duration 120
+```
 
+---
 ## **`help` Command**
 Displays help information for the NePS CLI, including a list of available commands and their descriptions.
 
@@ -163,3 +218,53 @@ Displays help information for the NePS CLI, including a list of available comman
 neps help --help
 ```
 
+---
+## **Using NePS as a State Machine**
+
+NePS can function as a state machine, allowing you to manually sample and report configurations using CLI commands. This is particularly useful in scenarios like architecture search, where different configurations may require varying computational resources. To utilize NePS in this manner, follow these steps:
+
+### **Step 1: Initialize and Configure `run_config.yaml**
+
+Begin by generating the `run_args` YAML configuration file. This file serves as the blueprint for your optimization experiments.
+
+
+```bash
+neps init
+```
+The `neps init` command creates run_config.yaml, which serves as the default configuration resource for all NePS commands.
+### **Step 2: Initialize the NePS Database**
+
+Set up the NePS database to enable the sampling and reporting of configurations via CLI commands.
+
+```bash
+neps init --database
+```
+This command initializes the NePS database, preparing the necessary folders and files required for managing your NePS run
+
+
+### **Step 3: Sample Configurations**
+
+Generate new configurations based on the existing NePS state. This step allows you to create configurations that you can manually evaluate.
+
+```bash
+neps sample-config --worker-id worker_1 --number-of-configs 5
+```
+
+- **`--worker_id worker_1`**: Identifies the worker responsible for sampling configurations.
+- **`--number-of-configs 5`**: Specifies the number of configurations to sample.
+
+### **Step 4: Evaluate and Report Configurations**
+
+After evaluating each sampled configuration, report its outcome to update the NePS state.
+
+```bash
+neps report-config 42 success --worker-id worker_1 --loss 0.95 --duration 120
+```
+
+- **`42`**: The ID of the trial being reported.
+- **`success`**: The outcome of the trial (`success`, `failed`, `crashed`).
+- **`--worker_id worker_1`**: Identifies the worker reporting the configuration.
+- **`--loss 0.95`**: The loss value obtained from the trial.
+- **`--duration 120`**: The duration of the evaluation in seconds.
+
+

neps/api.py

@@ -278,7 +278,7 @@ def _run_args(
             "mobster",
             "asha",
         ]
-        | BaseOptimizer
+        | BaseOptimizer | dict
     ) = "default",
     **searcher_kwargs,
 ) -> tuple[BaseOptimizer, dict]: