Benchmarking

[SRFU-NN]

Would it make sense to have a benchmarking example in the repo? Heavily inspired by the work @dk-teknologisk-mon did, but set up with the new model system.

I imagine it being a Jupyter notebook in design qualifications, which is set up to save the results in a json file.

If this is something we should do, let's discuss the details here, then I will write up a summary that we can discuss, and then we can start coding it.

[SRFU-NN]

Which models to use? I would suggest we start with the same that @dk-teknologisk-mon used. I can't find the list, so the following is made from my memory:

• Branin-Hoo
• Hart3
• Hart6

[SRFU-NN]

Which noise types do we want to investigate?

I am guessing constant noise is the default, since this is what we would naively expect to get from a physical system.

Proportional noise could also be interesting. Branin-Hoo and either Hart3 or Hart6 have different behaviours here that could both be interesting. Branin-Hoo would have lowest noise at the optima, while the harts would have the highest noise at the optima. I could see both happening with a physical system (the simplest example would be counting noise for either maximisation or minimisation problems), so I think we should include both in the benchmarking.

@RuneChristensen-NN suggested "reverse U shaped noise". If we are trying to optimise something that goes between 0 and 100%, we expect less noise in the ends, simply because there is less room for noise. I don't know which system we should use for this.

Another option is parameter dependent noise. Physical interpretations could be having two apparatuses with different responses and different noise, or different ingredients that cause different noise. Both would be dependent on a categorical variable. Would parameter dependent noise on a non-categorical variable make physical sense? Do we ever expect more (or less) noise at a higher temperature? When more of ingredient X is added?

[SRFU-NN]

What are the benchmarks we should investigate?

From @dk-teknologisk-mon 's benchmarking, the average amount of tests before we reach some defined level of quality.

But we are probably going to have to have several.