Low-dimensional representation of parameter space

[SRFU-NN]

If we have a parameter space where we assume there is a low-dimensional representation, we could build that into the kernel, and kind of do factor analysis and Bayesian optimisation at the same time.

This could be relevant for different solvents (they can probably be represented by 3 to 5 factors), different chromatographic columns (both GC and LC columns can be represented by a 5 factor model), or different ingredients (if we have different sweeteners, it probably makes sense to separate out a factor describing influence on sweetness and one describing influence on consistency).

Assume we have an isotropic, stationary kernel K(||x-x'||^2), say a Matérn kernel with all length scales at 1. We can then replace the Euclidian distance ||.|| = sqrt((x-x')T(x-x')) with a Mahalanobis distance d=sqrt((x-x')TM(x-x')), where M is a positive-definite covariance matrix. This makes the kernel K((x-x')TM(x-x')).

This requires some way to expose the entries of the matrix M as changeable hyperparameters, while ensuring that M is positive-definite.

If we know that the parameter space in D dimensions can be approximately represented in k dimensions (k<D), we can use M= Λ ΛT + Ψ, where Λ is a D × k matrix, and Ψ is a diagonal matrix (with positive entries) (see Rasmussen p. 89). With k=0, we reproduce the standard anisotropic Matérn kernel. For small k>0, we can capture low-dimensional phenomena.

With all entries in Ψ 0, we get what is effectively a linear constraint, e.g. sum constraint. We force D-k directions not to result in any change.

[SRFU-NN]

If we know something about the representation we want, we would inform the model by having priors on the hyperparameters, though that probably requires parametrizing the hyperparameters.

[SRFU-NN]

We could implement a (positive semi-definite) distance class, and let that be the input to a non-Euclidean_kernel, which applies an isotropic, stationary kernel, e.g. a Matérn kernel, to such a distance.

That way, we only need a rank 1 distance and a diagonal distance, and we can sum them appropriately to get any distance mentioned in this discussion.