Merge pull request #62 from automl/development

Development

README.md

@@ -1,16 +1,17 @@
-# HPOlib2
+# HPOBench
 
-HPOlib2 is a library for hyperparameter optimization and black-box optimization benchmark with a focus on reproducibility.
+HPOBench is a library for hyperparameter optimization and black-box optimization benchmark with a focus on reproducibility.
 
-**Note:** Hpolib2 is under active construction. Stay tuned for more benchmarks. Information on how to contribute a new benchmark will follow shortly.
+**Note:** HPOBench is under active construction. Stay tuned for more benchmarks. Information on how to contribute a new benchmark will follow shortly.
 
-**Note:** If you are looking for a different version of HPOlib2, you might be looking for [HPOlib1.5](https://github.com/automl/HPOlib1.5) 
+**Note:** "If you are looking for a different or older version of our benchmarking library, you might be looking for
+ [HPOlib1.5](https://github.com/automl/HPOlib1.5) 
 
 ## In 4 lines of code
 
 Run a random configuration within a singularity container
 ```python
-from hpolib.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
+from hpobench.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
 b = XGBoostBenchmark(task_id=167149, container_source='library://phmueller/automl', rng=1)
 config = b.get_configuration_space(seed=1).sample_configuration()
 result_dict = b.objective_function(configuration=config, fidelity={"n_estimators": 128, "subsample": 0.5}, rng=1)
@@ -19,7 +20,7 @@ result_dict = b.objective_function(configuration=config, fidelity={"n_estimators
 All benchmarks can also be queried with fewer or no fidelities:
 
 ```python
-from hpolib.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
+from hpobench.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
 b = XGBoostBenchmark(task_id=167149, container_source='library://phmueller/automl', rng=1)
 config = b.get_configuration_space(seed=1).sample_configuration()
 result_dict = b.objective_function(configuration=config, fidelity={"n_estimators": 128,}, rng=1)
@@ -35,7 +36,7 @@ Further requirements are: [ConfigSpace](https://github.com/automl/ConfigSpace),
 
 A simple example is the XGBoost benchmark which can be installed with `pip install .[xgboost]`
 ```python
-from hpolib.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
+from hpobench.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
 b = XGBoostBenchmark(task_id=167149)
 config = b.get_configuration_space(seed=1).sample_configuration()
 result_dict = b.objective_function(configuration=config, fidelity={"n_estimators": 128, "subsample": 0.5}, rng=1)
@@ -47,8 +48,8 @@ result_dict = b.objective_function(configuration=config, fidelity={"n_estimators
 Before we start, we recommend using a virtual environment. To run any benchmark using its singularity container, 
 run the following:
 ```
-git clone https://github.com/automl/HPOlib2.git
-cd HPOlib2 
+git clone https://github.com/automl/HPOBench.git
+cd HPOBench 
 pip install .
 ```
 
@@ -78,29 +79,29 @@ pip install .
 With singularity installed run the following to built the xgboost container
 
 ```bash
-cd hpolib/container/recipes/ml
+cd hpobench/container/recipes/ml
 sudo singularity build xgboost_benchmark Singularity.XGBoostBenchmark
 ```
 
 You can use this local image with:
 
 ```python
-from hpolib.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
+from hpobench.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
 b = XGBoostBenchmark(task_id=167149, container_name="xgboost_benchmark", 
-                     container_source='./') # path to hpolib/container/recipes/ml
+                     container_source='./') # path to hpobench/container/recipes/ml
 config = b.get_configuration_space(seed=1).sample_configuration()
 result_dict = b.objective_function(config, fidelity={"n_estimators": 128, "subsample": 0.5})
 ```
 
 ### Remove all caches
 
-#### HPOlib data
-HPOlib stores downloaded containers and datasets at the following locations:
+#### HPOBench data
+HPOBench stores downloaded containers and datasets at the following locations:
 
 ```bash
-$XDG_CONFIG_HOME # ~/.config/hpolib2
-$XDG_CACHE_HOME # ~/.cache/hpolib2
-$XDG_DATA_HOME # ~/.local/share/hpolib2
+$XDG_CONFIG_HOME # ~/.config/hpobench
+$XDG_CACHE_HOME # ~/.cache/hpobench
+$XDG_DATA_HOME # ~/.local/share/hpobench
 ```
 
 For crashes or when not properly shutting down containers, there might be socket files left under `/tmp/`.
@@ -127,10 +128,10 @@ See whether in `~/.singularity/instances/sing/$HOSTNAME/*/` there is a file that
 
 Status for Master Branch: 
 
-[![Build Status](https://travis-ci.org/automl/HPOlib2.svg?branch=master)](https://travis-ci.org/automl/HPOlib2)
-[![codecov](https://codecov.io/gh/automl/HPOlib2/branch/master/graph/badge.svg)](https://codecov.io/gh/automl/HPOlib2)
+[![Build Status](https://travis-ci.org/automl/HPOBench.svg?branch=master)](https://travis-ci.org/automl/HPOBench)
+[![codecov](https://codecov.io/gh/automl/HPOBench/branch/master/graph/badge.svg)](https://codecov.io/gh/automl/HPOBench)
 
 Status for Development Branch: 
 
-[![Build Status](https://travis-ci.org/automl/HPOlib2.svg?branch=development)](https://travis-ci.org/automl/HPOlib2)
-[![codecov](https://codecov.io/gh/automl/HPOlib2/branch/development/graph/badge.svg)](https://codecov.io/gh/automl/HPOlib2)
+[![Build Status](https://travis-ci.org/automl/HPOBench.svg?branch=development)](https://travis-ci.org/automl/HPOBench)
+[![codecov](https://codecov.io/gh/automl/HPOBench/branch/development/graph/badge.svg)](https://codecov.io/gh/automl/HPOBench)

changelog.md

@@ -1,3 +1,12 @@
+# 0.0.5
+  * Rename package to HPOBench
+  * Add BNN (pybnn) benchmark
+  * Update returned loss values for nasbench101 and tabular benchmarks
+  * Updat returned loss values for nasbench201 as well its data
+  * Nasbench201 is now 1 indexed instead of 0.
+  * Add MinMaxScaler to SVM Benchmark's preprocessing
+  * Add further tests
+
 # 0.0.4
   * improve test coverage
   * update HowToAddANewBenchmark.md

ci_scripts/codestyle.sh

@@ -3,15 +3,15 @@
 if [[ "$RUN_CODESTYLE" == "true" ]]; then
     echo "Performing codestyle checking"
 
-    test_codestyle=$(pycodestyle --max-line-length=120 ./hpolib)
+    test_codestyle=$(pycodestyle --max-line-length=120 ./hpobench)
     if [[ $test_codestyle ]]; then
       echo $test_codestyle
       exit 1
     else
       echo "Codesytle: No errors found"
     fi
 
-    test_flake=$(flake8 --max-line-length=120 ./hpolib)
+    test_flake=$(flake8 --max-line-length=120 ./hpobench)
     if [[ $test_flake ]]; then
       echo $test_flake
       exit 1

ci_scripts/install.sh

@@ -68,5 +68,5 @@ fi
 
 # remove the trailing comma
 install_packages="$(echo ${install_packages} | sed 's/,*\r*$//')"
-echo "Install HPOlib2 with options: ${install_packages}"
+echo "Install HPOBench with options: ${install_packages}"
 pip install .["${install_packages}"]

ci_scripts/script.sh

@@ -3,7 +3,7 @@
 if [[ "$RUN_TESTS" == "true" ]]; then
     if [[ "$RUN_CODECOV" == "true" ]]; then
         echo "Run tests with code coverage"
-        pytest -sv --cov=hpolib tests/
+        pytest -sv --cov=hpobench tests/
         exit_code=$?
 
         echo "Run code coverage"

codecov.yml

@@ -20,6 +20,6 @@ comment:
   require_changes: no
 
 ignore:
-  - "hpolib/benchmarks"
-  - "hpolib/util/dependencies.py"
-  - "hpolib/util/example_utils.py"
+  - "hpobench/benchmarks"
+  - "hpobench/util/dependencies.py"
+  - "hpobench/util/example_utils.py"

examples/container/tabular_benchmark_example.py

@@ -8,7 +8,7 @@
 
 container_source can be either a path to a registry (e.g. sylabs.io, singularity_hub.org) or a local path on your local
 file system. If it is a link to a registry, the container will be downloaded to the default data dir, set in the
-hpolibrc. A second call, will first look into the data directory, if the container is already available, so it will not
+hpobenchrc. A second call, will first look into the data directory, if the container is already available, so it will not
 be downloaded twice.
 
 Please install the necessary dependencies via ``pip install .`` and singularity (v3.5).
@@ -17,7 +17,7 @@
 
 import argparse
 
-from hpolib.container.benchmarks.nas.tabular_benchmarks import SliceLocalizationBenchmark as TabBenchmarkContainer
+from hpobench.container.benchmarks.nas.tabular_benchmarks import SliceLocalizationBenchmark as TabBenchmarkContainer
 
 
 def run_experiment(on_travis=False):

examples/container/xgboost_with_container.py

@@ -3,24 +3,24 @@
 ================================
 
 In this example, we show how to use a benchmark with a container. We provide container for some benchmarks.
-They are hosted on https://cloud.sylabs.io/library/keggensperger/automl.
+They are hosted on https://cloud.sylabs.io/library/muelleph/automl.
 
 Furthermore, we use different fidelities to train the xgboost model - the number of estimators as well as the fraction
 of training data points.
 
 To use the container-example, you have to have singulartiy (>3.5) installed. Follow the official installation guide on
 https://sylabs.io/guides/3.1/user-guide/quick_start.html#quick-installation-steps
 
-Furthermore, make sure to install the right dependencies for the hpolib via:
+Furthermore, make sure to install the right dependencies for the hpobench via:
 ``pip3 install .``.
 """
 
 import argparse
 import logging
 from time import time
 
-from hpolib.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark as Benchmark
-from hpolib.util.openml_data_manager import get_openmlcc18_taskids
+from hpobench.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark as Benchmark
+from hpobench.util.openml_data_manager import get_openmlcc18_taskids
 
 logging.basicConfig(level=logging.INFO)
 
@@ -66,7 +66,7 @@ def run_experiment(on_travis: bool = False):
 
 
 if __name__ == '__main__':
-    parser = argparse.ArgumentParser(prog='HPOlib CC Datasets', description='HPOlib2 on the CC18 data sets.',
+    parser = argparse.ArgumentParser(prog='HPOBench CC Datasets', description='HPOBench on the CC18 data sets.',
                                      usage='%(prog)s --array_id <task_id>')
 
     parser.add_argument('--on_travis', action='store_true',

examples/local/xgboost_local.py

@@ -10,8 +10,8 @@
 import argparse
 from time import time
 
-from hpolib.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark as Benchmark
-from hpolib.util.openml_data_manager import get_openmlcc18_taskids
+from hpobench.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark as Benchmark
+from hpobench.util.openml_data_manager import get_openmlcc18_taskids
 
 
 def run_experiment(on_travis: bool = False):
@@ -51,7 +51,7 @@ def run_experiment(on_travis: bool = False):
 
 
 if __name__ == '__main__':
-    parser = argparse.ArgumentParser(prog='HPOlib CC Datasets', description='HPOlib2 on the CC18 data sets.',
+    parser = argparse.ArgumentParser(prog='HPOBench CC Datasets', description='HPOBench on the CC18 data sets.',
                                      usage='%(prog)s --array_id <task_id>')
 
     parser.add_argument('--on_travis', action='store_true',

examples/w_optimizer/cartpole_bohb.py

@@ -18,9 +18,9 @@
 from hpbandster.core.worker import Worker
 from hpbandster.optimizers import BOHB
 
-from hpolib.container.benchmarks.rl.cartpole import CartpoleReduced as Benchmark
-from hpolib.util.example_utils import get_travis_settings, set_env_variables_to_use_only_one_core
-from hpolib.util.rng_helper import get_rng
+from hpobench.container.benchmarks.rl.cartpole import CartpoleReduced as Benchmark
+from hpobench.util.example_utils import get_travis_settings, set_env_variables_to_use_only_one_core
+from hpobench.util.rng_helper import get_rng
 
 logger = logging.getLogger('BOHB on cartpole')
 set_env_variables_to_use_only_one_core()
@@ -116,8 +116,8 @@ def run_experiment(out_path, on_travis):
 
 if __name__ == '__main__':
     import argparse
-    parser = argparse.ArgumentParser(prog='HPOlib - BOHB',
-                                     description='HPOlib2 with BOHB on Cartpole',
+    parser = argparse.ArgumentParser(prog='HPOBench - BOHB',
+                                     description='HPOBench with BOHB on Cartpole',
                                      usage='%(prog)s --out_path <string>')
     parser.add_argument('--out_path', default='./cartpole_smac_hb', type=str)
     parser.add_argument('--on_travis', action='store_true',

examples/w_optimizer/cartpole_hyperband.py

@@ -17,8 +17,8 @@
 from smac.intensification.hyperband import Hyperband
 from smac.scenario.scenario import Scenario
 
-from hpolib.container.benchmarks.rl.cartpole import CartpoleReduced as Benchmark
-from hpolib.util.example_utils import get_travis_settings, set_env_variables_to_use_only_one_core
+from hpobench.container.benchmarks.rl.cartpole import CartpoleReduced as Benchmark
+from hpobench.util.example_utils import get_travis_settings, set_env_variables_to_use_only_one_core
 
 logger = logging.getLogger("HB on cartpole")
 logging.basicConfig(level=logging.INFO)
@@ -93,8 +93,8 @@ def optimization_function_wrapper(cfg, seed, instance, budget):
 
 if __name__ == "__main__":
     import argparse
-    parser = argparse.ArgumentParser(prog='HPOlib - Hyperband',
-                                     description='HPOlib2 with HB on Cartpole',
+    parser = argparse.ArgumentParser(prog='HPOBench - Hyperband',
+                                     description='HPOBench with HB on Cartpole',
                                      usage='%(prog)s --out_path <string>')
     parser.add_argument('--out_path', default='./cartpole_smac_hb', type=str)
     parser.add_argument('--on_travis', action='store_true',

examples/w_optimizer/cartpole_succesive_halving.py

@@ -17,8 +17,8 @@
 from smac.intensification.hyperband import SuccessiveHalving
 from smac.scenario.scenario import Scenario
 
-from hpolib.container.benchmarks.rl.cartpole import CartpoleReduced as Benchmark
-from hpolib.util.example_utils import get_travis_settings, set_env_variables_to_use_only_one_core
+from hpobench.container.benchmarks.rl.cartpole import CartpoleReduced as Benchmark
+from hpobench.util.example_utils import get_travis_settings, set_env_variables_to_use_only_one_core
 
 logger = logging.getLogger("SH on cartpole")
 logging.basicConfig(level=logging.INFO)
@@ -96,8 +96,8 @@ def optimization_function_wrapper(cfg, seed, instance, budget):
 
 if __name__ == "__main__":
     import argparse
-    parser = argparse.ArgumentParser(prog='HPOlib - Successive Halving',
-                                     description='HPOlib2 with SH on Cartpole',
+    parser = argparse.ArgumentParser(prog='HPOBench - Successive Halving',
+                                     description='HPOBench with SH on Cartpole',
                                      usage='%(prog)s --out_path <string>')
     parser.add_argument('--out_path', default='./cartpole_smac_sh', type=str)
     parser.add_argument('--on_travis', action='store_true',

extra_requirements/pybnn.json

@@ -0,0 +1,4 @@
+{
+  "pybnn": ["Theano==1.0.5","sgmcmc@git+https://github.com/automl/sgmcmc#egg=sgmcmc-0.0.1",
+    "Lasagne@git+https://github.com/Lasagne/Lasagne#egg=Lasagne-0.0.1"]
+}

hpolib/HowToAddANewBenchmark.md → hpobench/HowToAddANewBenchmark.md

@@ -2,11 +2,11 @@
 
 ## Create a local benchmark
 
-Clone Hpolib2, switch to the development branch and create your own branch, then install hpolib2. 
+Clone HPOBench, switch to the development branch and create your own branch, then install hpobench. 
 with `pip install .`
 ```bash
-git clone https://github.com/automl/HPOlib2.git
-cd HPOlib2
+git clone https://github.com/automl/HPOBench.git
+cd HPOBench
 git checkout development
 git branch newBenchmark
 git checkout newBenchmark
@@ -15,21 +15,21 @@ pip install .
 
 Then: 
 
-  1. Implement your new benchmark `hpolib/benchmarks/<type>/<name>` inheriting from the base class 
-  `AbstractBenchmark` in `hpolib.abstract_benchmark`. Your benchmark should implement `__init__()`, 
+  1. Implement your new benchmark `hpobench/benchmarks/<type>/<name>` inheriting from the base class 
+  `AbstractBenchmark` in `hpobench.abstract_benchmark`. Your benchmark should implement `__init__()`, 
   `get_configuration_space()`, `get_fidelity_space()`, `objective_function()` and `objective_function_test()`.
-    A good example for this can be found in `hpolib/benchmarks/ml/xgboost_benchmark.py`
+    A good example for this can be found in `hpobench/benchmarks/ml/xgboost_benchmark.py`
   3. If your benchmarks needs a dataset (e.g. for training a ml model), please also implement a DataManager, see e.g.
-   `hpolib/util/openml_data_manager.py` with a `load()` method that downloads data once and reuses it for further calls.
+   `hpobench/util/openml_data_manager.py` with a `load()` method that downloads data once and reuses it for further calls.
   4. Collect all **additional Python** and **non-Python** dependencies while doing this. 
   Consider fixing the version of each dependency to maintain reproducibility.
   5. Add dependencies to PIPy in a new file to `/extra_requirements`
   6. Add the remaining dependencies or steps necessary to run your benchmark in the docstring of your benchmark class
-    (see, e.g. `hpolib/benchmarks/nas/nasbench_101.py`).
+    (see, e.g. `hpobench/benchmarks/nas/nasbench_101.py`).
   7. Verify that everything works with, e.g.
 
 ```python
-from hpolib.benchmarks.<type>.<newbenchmark> import <NewBenchmark>
+from hpobench.benchmarks.<type>.<newbenchmark> import <NewBenchmark>
 b = <NewBenchmark>(<some_args>, rng=1)
 config = b.get_configuration_space(seed=1).sample_configuration()
 result_dict = b.objective_function(configuration=config, rng=1)
@@ -42,21 +42,21 @@ Now, you can create a PR marked as [WIP] and proceed with building a containeriz
 
 ## Create a containerized benchmark
 
-  1. Create a container benchmark class in `hpolib/container/benchmarks/<type>/<name>` inheriting from the 
-  base class `AbstractBenchmarkClient` in `hpolib.container.client_abstract_benchmark`. 
-  Note: this are just a few lines of code, see, e.g. `hpolib/container/benchmarks/ml/xgboost_benchmark.py`)
-  2. Copy `hpolib/container/recipes/Singularity.template` to  `hpolib/container/recipes/<type>/name`
+  1. Create a container benchmark class in `hpobench/container/benchmarks/<type>/<name>` inheriting from the 
+  base class `AbstractBenchmarkClient` in `hpobench.container.client_abstract_benchmark`. 
+  Note: this are just a few lines of code, see, e.g. `hpobench/container/benchmarks/ml/xgboost_benchmark.py`)
+  2. Copy `hpobench/container/recipes/Singularity.template` to  `hpobench/container/recipes/<type>/name`
   3. Modify the recipe and add your **additional Python** and **non-Python** dependencies collected above. 
   3. Test your container locally (see below)
 
 Now, you can update your PR and let us know, so we can upload the container to Sylabs. Thanks.
 
 ## How to test your container locally
 
-  1. `cd hpolib/container/benchmarks/recipes/<type>` and change to following lines in the recipe:
+  1. `cd hpobench/container/benchmarks/recipes/<type>` and change to following lines in the recipe:
   ```bash
-    && git clone https://github.com/automl/HPOlib2.git \
-    && cd HPOlib2 \
+    && git clone https://github.com/automl/HPOBench.git \
+    && cd HPOBench \
     && git checkout development \
   ```
    to point to the branch/repo where your fork is on, e.g. `newBenchmark`.
@@ -65,7 +65,7 @@ Now, you can update your PR and let us know, so we can upload the container to S
   3. Verify that everything works with:
 
 ```python
-from hpolib.container.benchmarks.<type>.<newbenchmark> import <NewBenchmark>
+from hpobench.container.benchmarks.<type>.<newbenchmark> import <NewBenchmark>
 b = <NewBenchmark>(container_source="./", container_name="newBenchmark")
 res = b.objective_function(configuration=b.get_configuration_space(seed=1).sample_configuration())
 ```

hpobench/__init__.py

@@ -0,0 +1,4 @@
+__contact__ = "automl.org"
+
+from hpobench.__version__ import __version__  # noqa: F401
+from hpobench.config import config_file  # noqa: F401