From 049777e715c9cd103f871669024f645f92f172dd Mon Sep 17 00:00:00 2001 From: Janis Fix Date: Tue, 5 Mar 2024 10:24:57 +0100 Subject: [PATCH 1/2] Start working on doc enhancements --- docs/getting_started/ask_tell.md | 20 ++++++++++++++++++++ docs/index.md | 4 ++++ examples/00_interfacing_DEHB.ipynb | 2 +- mkdocs.yml | 1 + 4 files changed, 26 insertions(+), 1 deletion(-) create mode 100644 docs/getting_started/ask_tell.md diff --git a/docs/getting_started/ask_tell.md b/docs/getting_started/ask_tell.md new file mode 100644 index 0000000..4cb205b --- /dev/null +++ b/docs/getting_started/ask_tell.md @@ -0,0 +1,20 @@ +### Using the Ask & Tell interface +DEHB allows users to either utilize the Ask & Tell interface for manual task distribution or leverage the built-in functionality (`run`) to set up a Dask cluster autonomously. +The Ask & Tell functionality can be utilized as follows: +```python +optimizer = DEHB( + f=your_target_function, # Here we do not need to necessarily specify the target function, but it can still be useful to call 'run' later. + cs=config_space, + dimensions=dimensions, + min_fidelity=min_fidelity, + max_fidelity=max_fidelity) + +# Ask for next configuration to run +job_info = optimizer.ask() + +# Run the configuration for the given fidelity. Here you can freely distribute the computation to any worker you'd like. +result = your_target_function(config=job_info["config"], fidelity=job_info["fidelity"]) + +# When you received the result, feed them back to the optimizer +optimizer.tell(job_info, result) +``` \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index d244aeb..5fa3ae4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,7 @@ # Welcome to DEHB's documentation! ## Introduction +DEHB was designed to be an algorithm for Hyper Parameter Optimization (HPO). DEHB uses Differential Evolution (DE) under-the-hood as an Evolutionary Algorithm to power the black-box optimization that HPO problems pose. DE is a black-box optimization algorithm that generates candidate configurations $x$, to the black-box function $f(x)$, that is being optimized. The $x$ is evaluated by the black-box and the corresponding response $y$ is made available to the DE algorithm, which can then use this observation ($x$, $y$), along with previous such observations, to suggest a new candidate $x$ for the next evaluation. DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. `dehb` is a python package implementing the [DEHB](https://arxiv.org/abs/2105.09821) algorithm. It offers an intuitive interface to optimize user-defined problems using DEHB. @@ -24,6 +25,9 @@ pip install dehb pip install -e DEHB # -e stands for editable, lets you modify the code and rerun things ``` +## Using DEHB +DEHB allows users to either utilize the Ask & Tell interface for manual task distribution or leverage the built-in functionality (`run`) to set up a Dask cluster autonomously. Please refer to our [Getting Started](getting_started/single_worker.md) examples. + ## Contributing Please have a look at our [contributing guidelines](https://github.com/automl/DEHB/blob/master/CONTRIBUTING.md). diff --git a/examples/00_interfacing_DEHB.ipynb b/examples/00_interfacing_DEHB.ipynb index 49ad481..d13de07 100644 --- a/examples/00_interfacing_DEHB.ipynb +++ b/examples/00_interfacing_DEHB.ipynb @@ -39,7 +39,7 @@ "source": [ "DEHB was designed to be an algorithm for Hyper Parameter Optimization (HPO). DEHB uses Differential Evolution (DE) under-the-hood as an Evolutionary Algorithm to power the black-box optimization that HPO problems pose. DE is a black-box optimization algorithm that generates candidate configurations $x$, to the black-box function $f(x)$, that is being optimized. The $x$ is evaluated by the black-box and the corresponding response $y$ is made available to the DE algorithm, which can then use this observation ($x$, $y$), along with previous such observations, to suggest a new candidate $x$ for the next evaluation. \n", "\n", - "DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. Let $f(x)$ be the validation error of training a multilayer perceptron (MLP) on the complete training set. Multi-fidelity algorithms such as Hyperband, allow for cheaper approximations along a possible *fidelity*. For the MLP, a subset of the dataset maybe a cheaper approximation to the full data set evaluation. Whereas the fidelity can be quantifies as the fraction of the dataset used to evaluate the configuration $x$, instead of the full dataset. Such approximations can allow sneak-peek into the black-box, potentially revealing certain landscape feature of *f(x)*, thus rendering it a *gray*-box and not completely opaque and black! \n", + "DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. Let $f(x)$ be the validation error of training a multilayer perceptron (MLP) on the complete training set. Multi-fidelity algorithms such as Hyperband, allow for cheaper approximations along a possible *fidelity*. For the MLP, a subset of the dataset maybe a cheaper approximation to the full data set evaluation. Whereas the fidelity can be quantified as the fraction of the dataset used to evaluate the configuration $x$, instead of the full dataset. Such approximations can allow sneak-peek into the black-box, potentially revealing certain landscape features of *f(x)*, thus rendering it a *gray*-box and not completely opaque and black! \n", "\n", "The $z$ parameter is the fidelity parameter to the black-box function. If $z \\in [fidelity_{min}, fidelity_{max}]$, then $f(x, fidelity_{max})$ would be equivalent to the black-box case of $f(x)$.\n", "\n", diff --git a/mkdocs.yml b/mkdocs.yml index 8ee2ee9..27335f9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -5,6 +5,7 @@ nav: - Getting Started: - Single Worker: getting_started/single_worker.md - Parallel: getting_started/parallel.md + - Ask & Tell: getting_started/ask_tell.md - Examples: - Interfacing DEHB: examples/00_interfacing_DEHB.ipynb - Optimizing RandomForest using DEHB: examples/01.1_Optimizing_RandomForest_using_DEHB.ipynb From 4a7e915af8e3e5c4c53675305674701165b297da Mon Sep 17 00:00:00 2001 From: Bronzila Date: Wed, 6 Mar 2024 11:10:13 +0100 Subject: [PATCH 2/2] Proof-read docs and add feedback --- docs/getting_started/parallel.md | 10 ++--- docs/javascripts/katex.js | 10 +++++ docs/references/config_repository.md | 3 ++ examples/00_interfacing_DEHB.ipynb | 43 ++++++++----------- ...timizing_RandomForest_using_Ask_Tell.ipynb | 2 +- .../02_using_DEHB_without_ConfigSpace.ipynb | 4 +- mkdocs.yml | 12 ++++++ 7 files changed, 51 insertions(+), 33 deletions(-) create mode 100644 docs/javascripts/katex.js create mode 100644 docs/references/config_repository.md diff --git a/docs/getting_started/parallel.md b/docs/getting_started/parallel.md index d945eba..9d0f9ea 100644 --- a/docs/getting_started/parallel.md +++ b/docs/getting_started/parallel.md @@ -4,12 +4,12 @@ DEHB has been designed to interface a [Dask client](https://distributed.dask.org DEHB can either create a Dask client during instantiation and close/kill the client during garbage collection. Or a client can be passed as an argument during instantiation. -* Setting `n_workers` during instantiation \ - If set to `1` (default) then the entire process is a sequential run without invoking Dask. \ - If set to `>1` then a Dask Client is initialized with as many workers as `n_workers`. \ +* Setting `n_workers` during instantiation + If set to `1` (default) then the entire process is a sequential run without invoking Dask. + If set to `>1` then a Dask Client is initialized with as many workers as `n_workers`. This parameter is ignored if `client` is not None. -* Setting `client` during instantiation \ - When `None` (default), a Dask client is created using `n_workers` specified. \ +* Setting `client` during instantiation + When `None` (default), a Dask client is created using `n_workers` specified. Else, any custom-configured Dask Client can be created and passed as the `client` argument to DEHB. #### Using GPUs in a parallel run diff --git a/docs/javascripts/katex.js b/docs/javascripts/katex.js new file mode 100644 index 0000000..3828300 --- /dev/null +++ b/docs/javascripts/katex.js @@ -0,0 +1,10 @@ +document$.subscribe(({ body }) => { + renderMathInElement(body, { + delimiters: [ + { left: "$$", right: "$$", display: true }, + { left: "$", right: "$", display: false }, + { left: "\\(", right: "\\)", display: false }, + { left: "\\[", right: "\\]", display: true } + ], + }) + }) \ No newline at end of file diff --git a/docs/references/config_repository.md b/docs/references/config_repository.md new file mode 100644 index 0000000..21cefd9 --- /dev/null +++ b/docs/references/config_repository.md @@ -0,0 +1,3 @@ +# Configuration Repository + +::: dehb.utils.config_repository \ No newline at end of file diff --git a/examples/00_interfacing_DEHB.ipynb b/examples/00_interfacing_DEHB.ipynb index d13de07..093e860 100644 --- a/examples/00_interfacing_DEHB.ipynb +++ b/examples/00_interfacing_DEHB.ipynb @@ -5,7 +5,7 @@ "metadata": {}, "source": [ "# Interfacing DEHB\n", - "#### How to read this notebook\n", + "## How to read this notebook\n", "\n", "This notebook is designed to serve as a high-level, highly abstracted view of DEHB and how it can be used. The examples here are mere placeholders and *only* offer an interface to run DEHB on toy or actual problems.\n", "***" @@ -30,7 +30,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Getting started with DEHB" + "## Getting started with DEHB" ] }, { @@ -42,8 +42,8 @@ "DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. Let $f(x)$ be the validation error of training a multilayer perceptron (MLP) on the complete training set. Multi-fidelity algorithms such as Hyperband, allow for cheaper approximations along a possible *fidelity*. For the MLP, a subset of the dataset maybe a cheaper approximation to the full data set evaluation. Whereas the fidelity can be quantified as the fraction of the dataset used to evaluate the configuration $x$, instead of the full dataset. Such approximations can allow sneak-peek into the black-box, potentially revealing certain landscape features of *f(x)*, thus rendering it a *gray*-box and not completely opaque and black! \n", "\n", "The $z$ parameter is the fidelity parameter to the black-box function. If $z \\in [fidelity_{min}, fidelity_{max}]$, then $f(x, fidelity_{max})$ would be equivalent to the black-box case of $f(x)$.\n", - "\n", - "![boxes](imgs/black-gray-box.png)" + "\n", + "![boxes](https://github.com/automl/DEHB/blob/master/examples/imgs/black-gray-box.png?raw=true)" ] }, { @@ -52,7 +52,7 @@ "source": [ "HPO algorithms optimize such black/gray box by wrapping around this *target* function an interface, by which the algorithms can suggest new $x$ and also consume the result of the corresponding evaluation to store a collection of such ($x$, $y$) pairs. Therefore, to run DEHB, the most essential component required as input is the target function to optimize. Since DEHB can leverage a Hyperband, the target function interface should account for possible input of fidelity too. \n", "\n", - "### Sample interface for target function that DEHB optimizes" + "## Sample interface for target function that DEHB optimizes" ] }, { @@ -111,7 +111,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Defining a sample search space" + "## Defining a sample search space" ] }, { @@ -200,7 +200,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Defining fidelity range for the target function" + "## Defining fidelity range for the target function" ] }, { @@ -223,14 +223,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The above definitions are all the information that DEHB needs about a problem. We are now in a position to call upon DEHB and start running it, to tune $x$." + "The above definitions are all the information that DEHB needs about a problem. We are now in a position to call upon DEHB and start running it to tune $x$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Instantiating and running DEHB" + "## Instantiating and running DEHB" ] }, { @@ -271,7 +271,7 @@ "metadata": {}, "source": [ "DEHB allows the option of 3 different resources for its runtime budget:\n", - "#### 1) Running DEHB for a certain number of (successive halving) *brackets*" + "### 1) Running DEHB for a certain number of (successive halving) *brackets*" ] }, { @@ -298,7 +298,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### 2) Running DEHB for total number of *function evaluations*" + "### 2) Running DEHB for total number of *function evaluations*" ] }, { @@ -333,7 +333,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### 3) Running DEHB for total amount of *wallclock time*" + "### 3) Running DEHB for total amount of *wallclock time*" ] }, { @@ -452,28 +452,21 @@ "source": [ "***\n", "\n", - "### Conclusion\n", + "## Conclusion\n", "\n", "As detailed above, the problem definition needs to be input to DEHB as the following information:\n", - "* the *target_function* (`f`) that is the primary black-box function to optimize\n", - "* the fidelity range of `min_fidelity` and `max_fidelity` that allows the cheaper, faster gray-box optimization of `f`\n", - "* the search space or the input domain of the function `f`, that can be represented as a `ConfigSpace` object and passed to DEHB at initialization\n", + "* the *target_function* (`f`) that is the primary black-box function to optimize,\n", + "* the fidelity range of `min_fidelity` and `max_fidelity` that allows the cheaper, faster gray-box optimization of `f` and\n", + "* the search space or the input domain of the function `f`, that can be represented as a `ConfigSpace` object and passed to DEHB at initialization.\n", "\n", "\n", "Following which, DEHB can be run for any amount of practical real-world budget. It can be run for either:\n", "* a total amount of actual wallclock time, example one day (~86400 seconds), or\n", - "* a total number of function evaluations, or the number of times we want the black-box to be accessed for evaluation, across all fidelities\n", - "* the total number of *brackets* we want to run the DEHB algorithm for\n", + "* a total number of function evaluations, or the number of times we want the black-box to be accessed for evaluation, across all fidelities or\n", + "* the total number of *brackets* we want to run the DEHB algorithm for.\n", "\n", "DEHB will terminate once its chosen runtime budget is exhausted, and report the incumbent found. DEHB, as an *anytime* algorithm, constantly writes to disk a lightweight `json` file with the best found configuration and its score seen till that point." ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { diff --git a/examples/01.2_Optimizing_RandomForest_using_Ask_Tell.ipynb b/examples/01.2_Optimizing_RandomForest_using_Ask_Tell.ipynb index fb83f77..cf68add 100644 --- a/examples/01.2_Optimizing_RandomForest_using_Ask_Tell.ipynb +++ b/examples/01.2_Optimizing_RandomForest_using_Ask_Tell.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Using the Ask & Tell interface\n", + "# Optimizing RandomForest using the Ask & Tell interface\n", "\n", "This notebook aims to build on the template from `00_interfacing_DEHB` and use it on an actual problem, to optimize the hyperparameters of a Random Forest model, for a dataset. Here we use DEHB with the built-in ask and tell functionality.\n", "\n", diff --git a/examples/02_using_DEHB_without_ConfigSpace.ipynb b/examples/02_using_DEHB_without_ConfigSpace.ipynb index d943275..dd53994 100644 --- a/examples/02_using_DEHB_without_ConfigSpace.ipynb +++ b/examples/02_using_DEHB_without_ConfigSpace.ipynb @@ -35,7 +35,7 @@ "* `max_features` $-$ *float* $-$ [0.1, 0.9]\n", "* `min_samples_leaf` $-$ *integer* $-$ [1, 64] $-$ *log*-spacing \n", "\n", - "DE, and therefore DEHB, work in the unit hypercube space. The random individuals sampled at the beginning of DEHB, performs a uniform random sampling in the [0, 1] range for each parameter/dimension. Hence, each configuration suggested by DEHB also is in the [0, 1] range. The `vector_to_configspace` included in the DEHB source code, can reliably handle the transformation of the [0, 1] space of DEHB configurations to the original parameter space required. **In the absence of ConfigSpace usage, such a conversion needs to included as part of the objective/target function being passed.**" + "DE, and therefore DEHB, work in the unit hypercube space. The random individuals sampled at the beginning of DEHB, performs a uniform random sampling in the [0, 1] range for each parameter/dimension. Hence, each configuration suggested by DEHB also is in the [0, 1] range. The `vector_to_configspace` included in the DEHB source code, can reliably handle the transformation of the [0, 1] space of DEHB configurations to the original parameter space required. **In the absence of ConfigSpace usage, such a conversion needs to be included as part of the objective/target function being passed.**" ] }, { @@ -102,7 +102,7 @@ "**NOTE**: To handle categorical parameters would require custom representations for such cases. Categorical parameters don't have a lower or upper range but rather a possible list of discrete choices or values. Moreoever, categorical parameters can be string categories, boolean or even ordinal in nature.\n", "\n", "\n", - "Given this `transform_space` function, everything else from `01_Optimizing_RandomForest_using_DEHB` can be largely reused. Only the `target_function` needs to be modified to include the `transform_space` function. Also, the `configspace` parameter needs to be set tp `False` while initializing DEHB." + "Given this `transform_space` function, everything else from `01_Optimizing_RandomForest_using_DEHB` can be largely reused. Only the `target_function` needs to be modified to include the `transform_space` function. Also, the `configspace` parameter needs to be set to `False` while initializing DEHB." ] }, { diff --git a/mkdocs.yml b/mkdocs.yml index 27335f9..a6b7aa8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,4 +1,5 @@ site_name: DEHB +repo_url: https://github.com/automl/DEHB/ nav: - Home: index.md @@ -15,6 +16,7 @@ nav: - DEHB: references/dehb.md - DE: references/de.md - Bracket Manager: references/bracket_manager.md + - Configuration Repository: references/config_repository.md theme: name: material @@ -80,6 +82,8 @@ markdown_extensions: - name: mermaid class: mermaid format: !!python/name:pymdownx.superfences.fence_code_format + - pymdownx.arithmatex: + generic: true plugins: - search @@ -109,6 +113,14 @@ plugins: - mkdocs-jupyter: execute: true +extra_javascript: + - javascripts/katex.js + - https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.7/katex.min.js + - https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.7/contrib/auto-render.min.js + +extra_css: + - https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.7/katex.min.css + extra: version: provider: mike \ No newline at end of file