BOHB Tuner - support updating search space online

[pengyusi]

hi, i updated the parameter range in the terminal, the Web UI shows that the update is successful. but the neu trails started with the old parameter range.

here's what in the Web UI:
"noptepochs": {
"_type": "randint",
"_value": [
4,
29
]
here's the neuest trail:
Customized trial setting
ent_coef
0.0029863262812102388
n_steps
102
nminibatches
64
noptepochs
30
vf_coef
1
TRIAL_BUDGET
1

where the hyperparameter noptepochs is not right

[QuanluZhang]

@pengyusi which tuner did you use for this experiment?

[pengyusi]

@pengyusi which tuner did you use for this experiment?

BOHB

[QuanluZhang]

@pengyusi currently BOHB does not support updating search space online.

BTW, we should make it clear in doc that which tuner supports updating search space.

[pengyusi]

I've also tried the method you mentioned, after resume it still use the old range. So i stopped the experiment and started a new one.

maybe you can also give a feedback, if one updates the parameter range in terminal while using a BOHB tuner.

Thx for your very fest reply.

[QuanluZhang]

@pengyusi my mistake, the approach i just mentioned does not work.

Here is a workaround: export the finished trials in your previous experiment into a (json) file. then start a new experiment with the updated search space, and import the exported file to this experiment. Please refer to this page for the usage of nnictl experiment import. This helps the TPE part in BOHB get more data.

[pengyusi]

after importing the old experiment, i cann't see any trails of the old one in the Web UI. but the dispatcher logfile shows that the import is successful.

[02/25/2020, 09:24:06 PM] INFO (nni.msg_dispatcher_base/MainThread) Start dispatcher
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) start to handle_initialize
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 0 / 8
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 1 / 8
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 2 / 8
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 3 / 8
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 4 / 8
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 5 / 8
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 6 / 8
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 7 / 8
[02/25/2020, 09:24:06 PM] INFO (BOHB_Advisor/Thread-1) Successfully import tuning data to BOHB advisor.
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 0 / 8
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 1 / 8
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 2 / 8
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 3 / 8
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 4 / 8
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 5 / 8
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 6 / 8
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Importing data, current processing progress 7 / 8
[02/25/2020, 09:37:23 PM] INFO (BOHB_Advisor/Thread-1) Successfully import tuning data to BOHB advisor.

i'm not sure if the import is really successful.

[QuanluZhang]

@pengyusi yes, the data is imported successfully, because Tuner does not report error. We have not supported showing the imported data on webui yet, but this feature will be supported in future release.

[scarlett2018]

@pengyusi currently BOHB does not support updating search space online.

BTW, we should make it clear in doc that which tuner supports updating search space.

do we need documentation update? @QuanluZhang

[scarlett2018]

@pengyusi yes, the data is imported successfully, because Tuner does not report error. We have not supported showing the imported data on webui yet, but this feature will be supported in future release.

@Lijiaoa @yangou1988 @QuanluZhang - show we discuss the design for this later?