docs: readme

Author: be-marc

README.Rmd

@@ -3,8 +3,8 @@ output: github_document
 ---
 
 ```{r, include = FALSE}
-lgr::get_logger("mlr3")$set_threshold("warn")
-lgr::get_logger("bbotk")$set_threshold("warn")
+lgr::get_logger("mlr3")$set_threshold("error")
+lgr::get_logger("bbotk")$set_threshold("error")
 set.seed(1)
 options(datatable.print.class = FALSE, datatable.print.keys = FALSE, width = 200)
 ```
@@ -20,7 +20,19 @@ Package website: [release](https://mlr3hyperband.mlr-org.com/) | [dev](https://m
 [![Mattermost](https://img.shields.io/badge/chat-mattermost-orange.svg)](https://lmmisld-lmu-stats-slds.srv.mwn.de/mlr_invite/)
 <!-- badges: end -->
 
-This package provides hyperband tuning for [`mlr3`](https://mlr3.mlr-org.com).
+`mlr3hyperband` extends the [mlr3tuning](https://mlr3tuning.mlr-org.com/) package with multifidelity optimization methods based on the successive halving algorithm.
+It currently provides the following optimizers for [bbotk](https://bbotk.mlr-org.com/) and tuner for [mlr3tuning](https://mlr3tuning.mlr-org.com/):
+
+* Successive Halving (`OptimizerSuccessiveHalving` & `TunerSuccessiveHalving`)
+* Hyperband (`OptimizerSuccessiveHalving` & `TunerSuccessiveHalving`)
+
+## Resources
+
+* mlr3book chapter on [hyperband](https://mlr3book.mlr-org.com/optimization.html#hyperband)
+and [hyperparameter tuning](https://mlr3book.mlr-org.com/optimization.html#tuning).
+* The original publications introducing the [successive halving](https://arxiv.org/abs/1502.07943) and
+[hyperband](https://arxiv.org/abs/1603.06560).
+* Ask questions on [Stackoverflow (tag #mlr3)](https://stackoverflow.com/questions/tagged/mlr3)
 
 ## Installation
 
@@ -36,26 +48,10 @@ Install the development version from GitHub:
 remotes::install_github("mlr-org/mlr3hyperband")
 ```
 
-## Resources
-
-* mlr3book chapter on [hyperband](https://mlr3book.mlr-org.com/optimization.html#hyperband)
-and [hyperparameter tuning](https://mlr3book.mlr-org.com/optimization.html#tuning).
-* The original [paper](https://arxiv.org/abs/1603.06560) introducing the hyperband algorithm.
-
-## Hyperband
-
-Hyperband is a budget oriented-procedure, weeding out suboptimally performing configurations early on during their training process aiming at increasing the efficiency of the tuning procedure.
-For this, several brackets are constructed with an associated set of configurations for each bracket.
-These configuration are initialized by stochastic, often uniform, sampling.
-Each bracket is divided into multiple stages, and configurations are evaluated for a increasing budget in each stage.
-Note that currently all configurations are trained completely from the beginning, so no online updates to the models are performed.
-
-Different brackets are initialized with different number of configurations, and different budget sizes.
-To identify the budget for evaluating hyperband, the user has to specify explicitly which hyperparameter of the learner influences the budget by tagging a single hyperparameter in the parameter set with `"budget"`.
-An alternative approach using subsampling and pipelines is described further below.
-
 ## Examples
 
+### Basic
+
 ```{r, include = FALSE}
 # mute load messages
 library(bbotk)
@@ -64,53 +60,24 @@ library(mlr3hyperband)
 library(mlr3learners)
 ```
 
-### Basic
-
-If you are already familiar with `mlr3tuning`, then the only change compared to other tuners is to give a numeric hyperparameter a `budget` tag.
-Afterwards, you can handle hyperband like all other tuners.
-Originally, hyperband was created with a “natural” learning parameter as the budget parameter in mind, like `nrounds` of the XGBoost learner.
-
 ```{r}
-library(mlr3verse)
 library(mlr3hyperband)
 library(mlr3learners)
 
-# define hyperparameter and budget parameter
-search_space = ps(
-  nrounds = p_int(lower = 1, upper = 16, tags = "budget"),
-  eta = p_dbl(lower = 0, upper = 1),
-  booster = p_fct(levels = c("gbtree", "gblinear", "dart"))
+# load learner, define search space and tag budget hyperparameter
+learner = lrn("classif.xgboost",
+  nrounds           = to_tune(p_int(27, 243, tags = "budget")),
+  eta               = to_tune(1e-4, 1, logscale = TRUE),
+  max_depth         = to_tune(1, 20),
+  colsample_bytree  = to_tune(1e-1, 1),
+  colsample_bylevel = to_tune(1e-1, 1),
+  lambda            = to_tune(1e-3, 1e3, logscale = TRUE),
+  alpha             = to_tune(1e-3, 1e3, logscale = TRUE),
+  subsample         = to_tune(1e-1, 1)
 )
 
-# hyperparameter tuning on the pima indians diabetes data set
-instance = tune(
-  method = "hyperband",
-  task = tsk("pima"),
-  learner = lrn("classif.xgboost", eval_metric = "logloss"),
-  resampling = rsmp("cv", folds = 3),
-  measures = msr("classif.ce"),
-  search_space = search_space
-)
-
-# best performing hyperparameter configuration
-instance$result
-```
-
-### Subsampling
-
-Additionally, it is also possible to use `mlr3hyperband` to tune learners that do not have a natural fidelity parameter.
-In such a case `mlr3pipelines` can be used to define data subsampling as a preprocessing step.
-Then, the `frac` parameter of subsampling, defining the fraction of the training data to be used, can act as the budget parameter.
-
-```{r}
-learner = po("subsample") %>>% lrn("classif.rpart")
-
-# define subsampling parameter as budget
-search_space = ps(
-  classif.rpart.cp = p_dbl(lower = 0.001, upper = 0.1),
-  classif.rpart.minsplit = p_int(lower = 1, upper = 10),
-  subsample.frac = p_dbl(lower = 0.1, upper = 1, tags = "budget")
-)
+# set parallel backend
+future::plan("multisession")
 
 # hyperparameter tuning on the pima indians diabetes data set
 instance = tune(
@@ -119,38 +86,42 @@ instance = tune(
   learner = learner,
   resampling = rsmp("cv", folds = 3),
   measures = msr("classif.ce"),
-  search_space = search_space
+  eta = 3
 )
-
-# best performing hyperparameter configuration
-instance$result
 ```
 
-### Successive Halving
+### Subsample
 
-```{r, message = FALSE}
+```{r}
 library(mlr3hyperband)
 library(mlr3learners)
+library(mlr3pipelines)
 
-# define hyperparameter and budget parameter
-search_space = ps(
-  nrounds = p_int(lower = 1, upper = 16, tags = "budget"),
-  eta = p_dbl(lower = 0, upper = 1),
-  booster = p_fct(levels = c("gbtree", "gblinear", "dart"))
+# load learner and define search space
+learner = lrn("classif.rpart",
+  minsplit  = to_tune(2, 128, logscale = TRUE),
+  minbucket = to_tune(1, 64, logscale = TRUE),
+  cp        = to_tune(1e-04, 1e-1, logscale = TRUE)
 )
 
-# hyperparameter tuning on the pima indians diabetes data set
+# create graph learner with subsampling
+graph_learner = as_learner(po("subsample") %>>% learner)
+
+# tag budget parameter
+graph_learner$param_set$values$subsample.frac = to_tune(p_dbl(3^-3, 1, tags = "budget"))
+
+# set parallel backend
+future::plan("multisession")
+
+# hyperparameter tuning on the spam data set
 instance = tune(
-  method = "successive_halving",
-  task = tsk("pima"),
-  learner = lrn("classif.xgboost", eval_metric = "logloss"),
+  method = "hyperband",
+  task = tsk("spam"),
+  learner = graph_learner,
   resampling = rsmp("cv", folds = 3),
   measures = msr("classif.ce"),
-  search_space = search_space
+  eta = 3
 )
-
-# best performing hyperparameter configuration
-instance$result
 ```
 
 ### Quick general-purpose optimization
@@ -161,9 +132,9 @@ library(mlr3hyperband)
 
 # define hyperparameter and budget parameter
 search_space = domain = ps(
-  x1 = p_dbl(-5, 10),
-  x2 = p_dbl(0, 15),
-  fidelity = p_dbl(1e-2, 1, tags = "budget")
+  x1        = p_dbl(-5, 10),
+  x2        = p_dbl(0, 15),
+  fidelity  = p_dbl(1e-2, 1, tags = "budget")
 )
 
 # modified branin function
@@ -174,7 +145,7 @@ objective = ObjectiveRFun$new(
 )
 
 # optimize branin function with hyperband
-result = bb_optimize(objective, method = "hyperband", search_space = search_space, term_evals = NULL)
+result = bb_optimize(objective, method = "hyperband", search_space = search_space, term_evals = NULL, eta = 2)
 
 # optimized parameters
 result$par