Creating new Info instances before using tell

[LukasFehring]

• HyperSweeper version: 0.1.0
• Python version: Any
• Operating System: Any

Description

Before utilizing tell, the software creates a new Info instance, thereby possibly using different values, than used originally. This can cause issues in smac

    logged_performance = -performance if self.maximize else performance
    info = Info(budget=budget, seed=seed, config=config)
    value = Result(performance=logged_performance, cost=cost)
    self.optimizer.tell(info=info, value=value)

Additional Info

I would recommend saving the info instances and using them to process tell.

[TheEimer]

I agree, it's easier to use the saved ones, but why does it cause issues? Is SMAC so sensitive that e.g. rounding errors make a difference?

[LukasFehring]

Short answer: Yes.

Long answer: In this situation

# %%
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
import logging
import os
import math
import time
import json
import pickle
import numpy as np

from ConfigSpace.hyperparameters import (
    CategoricalHyperparameter,
    NormalIntegerHyperparameter,
    OrdinalHyperparameter,
    UniformIntegerHyperparameter,
)
from py_experimenter.result_processor import ResultProcessor
from hydra.utils import to_absolute_path

# from deepcave import Recorder, Objective
from smac import HyperparameterOptimizationFacade, Scenario
from smac.intensifier.hyperband import Hyperband
from smac.runhistory.dataclasses import TrialValue, TrialInfo
import pickle

# %%
with open("minimal_working_example/scenario.pkl", "rb") as file:
    scenario=pickle.load(file)

# %%
scenario

# %%
# Kann es wegen dem budget kaputt gehen?

# %%
intensifier = HyperparameterOptimizationFacade.get_intensifier(
                scenario,
                max_config_calls=1, # TODO is this an issue?
            )

# %%
def dummy(arg, seed, budget):
            return -1

# %%
smac = HyperparameterOptimizationFacade(
    scenario,
    dummy,
    intensifier=intensifier,
    overwrite=True
)

# %%
config = smac.ask()
info = TrialInfo(budget=100, seed=0.0, config=config.config)
value = TrialValue(cost=float(-1.0))
smac.tell(info=info, value=value)

# %%
info

# %%
new_config = smac.ask()

The smac.ask hits an infinite loop. I do not understand the details of smac, but by reusing the same info object this could be solved

[TheEimer]

Now infos aren't parsed anymore, does it work now?

[LukasFehring]

I am not using the current hypersweeper version, but will look into it, when adding the py-experimenter :D