Refactor Space

[SRFU-NN]

There is a lot simplicity to be gained by a refactor of the space. The general idea is:

• Make a Space abstract class, and define needed properties (dimension, …) and methods (sample_point, distance, kernel, contains, …). Space.sample_point should take a list that has the same length as dimension. That way, we can do structured sampling (LHS, QRS) regardless of the structure.
• Define the Spaces with dimension 1: Categorical, Integer and Real. These are equivalent to the present Dimensions.
• Make a ProductSpace, which takes a list of Spaces, and produces the product Space. This is equivalent to the current Space. Its dimension is equal to the sum of the dimensions of the spaces in the list. Its kernel is the product of the kernels of the individual sub-spaces.
• Define a number of ConstraintSpaces, that each take a Space, and only allows for some points. Its dimension is equal to that of the underlying Space. sample_point needs to be carefully considered.

SumEqual would be a special case, since its apparent dimensionality is different from its true dimensionality (however those concepts are defined). sample_point becomes much easier here, you just need a area (volume, ...) preserving function, and transform with that (see what Martin Roberts did for sampling on a sphere).

Since kernel is a property of each dimension, it can be freely changed per dimension. We need to expose the hyperparameters, but there is a base class for this in sci-kit learn.

We risk losing the ability to calculate the gradient of the kernel in some cases (Categorical, Integer), so we would have to have an alternative optimisation strategy. I suggest evolutionary: Make a population of points, evaluate them on the surrogate function (or is it on the acquisition function?), choose the best X, cross-breed and mutate those, and continue. There are packages that does this, we already do it for NSGA-II.

For the spaces where we can calculate the gradient of the kernel, we could have addition (go X distance in direction Y from point Z), which would allow use to have more classical optimisation, even on weird spaces. We have to handle going "over the edge", but we already do this for normal dimensions if we go beyond the borders.

What we gain is that all of the if, then, else lines of Optimizer just goes away. It just needs to know the dimensionality of the space and whether it can ask for the gradient of the kernel, and then let the space deal with the rest. This will make it much easier to maintain, and easier to read. It will potentially be harder to understand what it does where, though.