Merge pull request #76 from automl/development

Doc updates

docs/getting_started/ask_tell.md

@@ -0,0 +1,20 @@
+### Using the Ask & Tell interface
+DEHB allows users to either utilize the Ask & Tell interface for manual task distribution or leverage the built-in functionality (`run`) to set up a Dask cluster autonomously.
+The Ask & Tell functionality can be utilized as follows:
+```python
+optimizer = DEHB(
+    f=your_target_function, # Here we do not need to necessarily specify the target function, but it can still be useful to call 'run' later.
+    cs=config_space, 
+    dimensions=dimensions, 
+    min_fidelity=min_fidelity, 
+    max_fidelity=max_fidelity)
+
+# Ask for next configuration to run
+job_info = optimizer.ask()
+
+# Run the configuration for the given fidelity. Here you can freely distribute the computation to any worker you'd like.
+result = your_target_function(config=job_info["config"], fidelity=job_info["fidelity"])
+
+# When you received the result, feed them back to the optimizer
+optimizer.tell(job_info, result)
+```

docs/getting_started/parallel.md

@@ -4,12 +4,12 @@ DEHB has been designed to interface a [Dask client](https://distributed.dask.org
 DEHB can either create a Dask client during instantiation and close/kill the client during garbage collection. 
 Or a client can be passed as an argument during instantiation.
 
-* Setting `n_workers` during instantiation \
-    If set to `1` (default) then the entire process is a sequential run without invoking Dask. \
-    If set to `>1` then a Dask Client is initialized with as many workers as `n_workers`. \
+* Setting `n_workers` during instantiation  
+    If set to `1` (default) then the entire process is a sequential run without invoking Dask.  
+    If set to `>1` then a Dask Client is initialized with as many workers as `n_workers`.  
     This parameter is ignored if `client` is not None.
-* Setting `client` during instantiation \
-    When `None` (default), a Dask client is created using `n_workers` specified. \
+* Setting `client` during instantiation  
+    When `None` (default), a Dask client is created using `n_workers` specified.  
     Else, any custom-configured Dask Client can be created and passed as the `client` argument to DEHB.
 
 #### Using GPUs in a parallel run

docs/index.md

@@ -1,6 +1,7 @@
 # Welcome to DEHB's documentation!
 
 ## Introduction
+DEHB was designed to be an algorithm for Hyper Parameter Optimization (HPO). DEHB uses Differential Evolution (DE) under-the-hood as an Evolutionary Algorithm to power the black-box optimization that HPO problems pose. DE is a black-box optimization algorithm that generates candidate configurations $x$, to the black-box function $f(x)$, that is being optimized. The $x$ is evaluated by the black-box and the corresponding response $y$ is made available to the DE algorithm, which can then use this observation ($x$, $y$), along with previous such observations, to suggest a new candidate $x$ for the next evaluation. DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$.
 
 `dehb` is a python package implementing the [DEHB](https://arxiv.org/abs/2105.09821) algorithm. It offers an intuitive interface to optimize user-defined problems using DEHB.
 
@@ -24,6 +25,9 @@ pip install dehb
     pip install -e DEHB  # -e stands for editable, lets you modify the code and rerun things
     ```
 
+## Using DEHB
+DEHB allows users to either utilize the Ask & Tell interface for manual task distribution or leverage the built-in functionality (`run`) to set up a Dask cluster autonomously. Please refer to our [Getting Started](getting_started/single_worker.md) examples.
+
 ## Contributing
 Please have a look at our [contributing guidelines](https://github.com/automl/DEHB/blob/master/CONTRIBUTING.md).
 

docs/javascripts/katex.js

@@ -0,0 +1,10 @@
+document$.subscribe(({ body }) => { 
+    renderMathInElement(body, {
+      delimiters: [
+        { left: "$$",  right: "$$",  display: true },
+        { left: "$",   right: "$",   display: false },
+        { left: "\\(", right: "\\)", display: false },
+        { left: "\\[", right: "\\]", display: true }
+      ],
+    })
+  })

docs/references/config_repository.md

@@ -0,0 +1,3 @@
+# Configuration Repository
+
+::: dehb.utils.config_repository

examples/00_interfacing_DEHB.ipynb

@@ -5,7 +5,7 @@
    "metadata": {},
    "source": [
     "# Interfacing DEHB\n",
-    "#### How to read this notebook\n",
+    "## How to read this notebook\n",
     "\n",
     "This notebook is designed to serve as a high-level, highly abstracted view of DEHB and how it can be used. The examples here are mere placeholders and *only* offer an interface to run DEHB on toy or actual problems.\n",
     "***"
@@ -30,7 +30,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Getting started with DEHB"
+    "## Getting started with DEHB"
    ]
   },
   {
@@ -39,11 +39,11 @@
    "source": [
     "DEHB was designed to be an algorithm for Hyper Parameter Optimization (HPO). DEHB uses Differential Evolution (DE) under-the-hood as an Evolutionary Algorithm to power the black-box optimization that HPO problems pose. DE is a black-box optimization algorithm that generates candidate configurations $x$, to the black-box function $f(x)$, that is being optimized. The $x$ is evaluated by the black-box and the corresponding response $y$ is made available to the DE algorithm, which can then use this observation ($x$, $y$), along with previous such observations, to suggest a new candidate $x$ for the next evaluation. \n",
     "\n",
-    "DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. Let $f(x)$ be the validation error of training a multilayer perceptron (MLP) on the complete training set. Multi-fidelity algorithms such as Hyperband, allow for cheaper approximations along a possible *fidelity*. For the MLP, a subset of the dataset maybe a cheaper approximation to the full data set evaluation. Whereas the fidelity can be quantifies as the fraction of the dataset used to evaluate the configuration $x$, instead of the full dataset. Such approximations can allow sneak-peek into the black-box, potentially revealing certain landscape feature of *f(x)*, thus rendering it a *gray*-box and not completely opaque and black! \n",
+    "DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. Let $f(x)$ be the validation error of training a multilayer perceptron (MLP) on the complete training set. Multi-fidelity algorithms such as Hyperband, allow for cheaper approximations along a possible *fidelity*. For the MLP, a subset of the dataset maybe a cheaper approximation to the full data set evaluation. Whereas the fidelity can be quantified as the fraction of the dataset used to evaluate the configuration $x$, instead of the full dataset. Such approximations can allow sneak-peek into the black-box, potentially revealing certain landscape features of *f(x)*, thus rendering it a *gray*-box and not completely opaque and black! \n",
     "\n",
     "The $z$ parameter is the fidelity parameter to the black-box function. If $z \\in [fidelity_{min}, fidelity_{max}]$, then $f(x, fidelity_{max})$ would be equivalent to the black-box case of $f(x)$.\n",
-    "\n",
-    "![boxes](imgs/black-gray-box.png)"
+    "<!--- Here we need to use the weblink in order to correctly show the image in the docs--->\n",
+    "![boxes](https://github.com/automl/DEHB/blob/master/examples/imgs/black-gray-box.png?raw=true)"
    ]
   },
   {
@@ -52,7 +52,7 @@
    "source": [
     "HPO algorithms optimize such black/gray box by wrapping around this *target* function an interface, by which the algorithms can suggest new $x$ and also consume the result of the corresponding evaluation to store a collection of such ($x$, $y$) pairs. Therefore, to run DEHB, the most essential component required as input is the target function to optimize. Since DEHB can leverage a Hyperband, the target function interface should account for possible input of fidelity too. \n",
     "\n",
-    "### Sample interface for target function that DEHB optimizes"
+    "## Sample interface for target function that DEHB optimizes"
    ]
   },
   {
@@ -111,7 +111,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Defining a sample search space"
+    "## Defining a sample search space"
    ]
   },
   {
@@ -200,7 +200,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Defining fidelity range for the target function"
+    "## Defining fidelity range for the target function"
    ]
   },
   {
@@ -223,14 +223,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The above definitions are all the information that DEHB needs about a problem. We are now in a position to call upon DEHB and start running it, to tune $x$."
+    "The above definitions are all the information that DEHB needs about a problem. We are now in a position to call upon DEHB and start running it to tune $x$."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Instantiating and running DEHB"
+    "## Instantiating and running DEHB"
    ]
   },
   {
@@ -271,7 +271,7 @@
    "metadata": {},
    "source": [
     "DEHB allows the option of 3 different resources for its runtime budget:\n",
-    "#### 1) Running DEHB for a certain number of (successive halving) *brackets*"
+    "### 1) Running DEHB for a certain number of (successive halving) *brackets*"
    ]
   },
   {
@@ -298,7 +298,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### 2) Running DEHB for total number of *function evaluations*"
+    "### 2) Running DEHB for total number of *function evaluations*"
    ]
   },
   {
@@ -333,7 +333,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### 3) Running DEHB for total amount of *wallclock time*"
+    "### 3) Running DEHB for total amount of *wallclock time*"
    ]
   },
   {
@@ -452,28 +452,21 @@
    "source": [
     "***\n",
     "\n",
-    "### Conclusion\n",
+    "## Conclusion\n",
     "\n",
     "As detailed above, the problem definition needs to be input to DEHB as the following information:\n",
-    "* the *target_function* (`f`) that is the primary black-box function to optimize\n",
-    "* the fidelity range of `min_fidelity` and `max_fidelity` that allows the cheaper, faster gray-box optimization of `f`\n",
-    "* the search space or the input domain of the function `f`, that can be represented as a `ConfigSpace` object and passed to DEHB at initialization\n",
+    "* the *target_function* (`f`) that is the primary black-box function to optimize,\n",
+    "* the fidelity range of `min_fidelity` and `max_fidelity` that allows the cheaper, faster gray-box optimization of `f` and\n",
+    "* the search space or the input domain of the function `f`, that can be represented as a `ConfigSpace` object and passed to DEHB at initialization.\n",
     "\n",
     "\n",
     "Following which, DEHB can be run for any amount of practical real-world budget. It can be run for either:\n",
     "* a total amount of actual wallclock time, example one day (~86400 seconds), or\n",
-    "* a total number of function evaluations, or the number of times we want the black-box to be accessed for evaluation, across all fidelities\n",
-    "* the total number of *brackets* we want to run the DEHB algorithm for\n",
+    "* a total number of function evaluations, or the number of times we want the black-box to be accessed for evaluation, across all fidelities or\n",
+    "* the total number of *brackets* we want to run the DEHB algorithm for.\n",
     "\n",
     "DEHB will terminate once its chosen runtime budget is exhausted, and report the incumbent found. DEHB, as an *anytime* algorithm, constantly writes to disk a lightweight `json` file with the best found configuration and its score seen till that point."
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

examples/01.2_Optimizing_RandomForest_using_Ask_Tell.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Using the Ask & Tell interface\n",
+    "# Optimizing RandomForest using the Ask & Tell interface\n",
     "\n",
     "This notebook aims to build on the template from `00_interfacing_DEHB` and use it on an actual problem, to optimize the hyperparameters of a Random Forest model, for a dataset. Here we use DEHB with the built-in ask and tell functionality.\n",
     "\n",

examples/02_using_DEHB_without_ConfigSpace.ipynb

@@ -35,7 +35,7 @@
     "* `max_features` $-$ *float* $-$ [0.1, 0.9]\n",
     "* `min_samples_leaf` $-$ *integer* $-$ [1, 64] $-$ *log*-spacing \n",
     "\n",
-    "DE, and therefore DEHB, work in the unit hypercube space. The random individuals sampled at the beginning of DEHB, performs a uniform random sampling in the [0, 1] range for each parameter/dimension. Hence, each configuration suggested by DEHB also is in the [0, 1] range. The `vector_to_configspace` included in the DEHB source code, can reliably handle the transformation of the [0, 1] space of DEHB configurations to the original parameter space required. **In the absence of ConfigSpace usage, such a conversion needs to included as part of the objective/target function being passed.**"
+    "DE, and therefore DEHB, work in the unit hypercube space. The random individuals sampled at the beginning of DEHB, performs a uniform random sampling in the [0, 1] range for each parameter/dimension. Hence, each configuration suggested by DEHB also is in the [0, 1] range. The `vector_to_configspace` included in the DEHB source code, can reliably handle the transformation of the [0, 1] space of DEHB configurations to the original parameter space required. **In the absence of ConfigSpace usage, such a conversion needs to be included as part of the objective/target function being passed.**"
    ]
   },
   {
@@ -102,7 +102,7 @@
     "**NOTE**: To handle categorical parameters would require custom representations for such cases. Categorical parameters don't have a lower or upper range but rather a possible list of discrete choices or values. Moreoever, categorical parameters can be string categories, boolean or even ordinal in nature.\n",
     "\n",
     "\n",
-    "Given this `transform_space` function, everything else from `01_Optimizing_RandomForest_using_DEHB` can be largely reused. Only the `target_function` needs to be modified to include the `transform_space` function. Also, the `configspace` parameter needs to be set tp `False` while initializing DEHB."
+    "Given this `transform_space` function, everything else from `01_Optimizing_RandomForest_using_DEHB` can be largely reused. Only the `target_function` needs to be modified to include the `transform_space` function. Also, the `configspace` parameter needs to be set to `False` while initializing DEHB."
    ]
   },
   {

mkdocs.yml

@@ -1,10 +1,12 @@
 site_name: DEHB
+repo_url: https://github.com/automl/DEHB/
 
 nav:
   - Home: index.md
   - Getting Started:
     - Single Worker: getting_started/single_worker.md
     - Parallel: getting_started/parallel.md
+    - Ask & Tell: getting_started/ask_tell.md
   - Examples:
     - Interfacing DEHB: examples/00_interfacing_DEHB.ipynb
     - Optimizing RandomForest using DEHB: examples/01.1_Optimizing_RandomForest_using_DEHB.ipynb
@@ -14,6 +16,7 @@ nav:
     - DEHB: references/dehb.md
     - DE: references/de.md
     - Bracket Manager: references/bracket_manager.md
+    - Configuration Repository: references/config_repository.md
 
 theme:
   name: material
@@ -79,6 +82,8 @@ markdown_extensions:
       - name: mermaid
         class: mermaid
         format: !!python/name:pymdownx.superfences.fence_code_format
+  - pymdownx.arithmatex:
+      generic: true
 
 plugins:
   - search
@@ -108,6 +113,14 @@ plugins:
   - mkdocs-jupyter:
       execute: true
 
+extra_javascript:
+  - javascripts/katex.js
+  - https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.7/katex.min.js
+  - https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.7/contrib/auto-render.min.js
+
+extra_css:
+  - https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.7/katex.min.css
+
 extra:
   version:
     provider: mike