Write Tune step

[kordc]

def do(self, previous_states: Dict):
        trainer = Trainer()  # Here we should write other object for this.
        # TODO how to solve this?
        trainer.tune(model=self.model, datamodule=self.datamodule)

[kordc]

Tune capability:

• random over grid search. For simultaneously tuning multiple hyperparameters it may sound tempting to use grid search to ensure coverage of all settings, but keep in mind that it is best to use random search instead. Intuitively, this is because neural nets are often much more sensitive to some parameters than others. In the limit, if a parameter a matters but changing b has no effect then you’d rather sample a more throughly than at a few fixed points multiple times.
• hyper-parameter optimization. There is a large number of fancy bayesian hyper-parameter optimization toolboxes around and a few of my friends have also reported success with them, but my personal experience is that the state of the art approach to exploring a nice and wide space of models and hyperparameters is to use an intern :). Just kidding.

[SebChw]

I think that first version of Tune should just support the random search. Over parameters/data/model modifiers

[kordc]

I'd then name it a RandomSearch step. I agree

[SebChw]

Probably right we shouldn't have one Tune step with options but rather few simpler ones.

[trebacz626]

Also we need to remember that trainer.tune tunes only learning_rate