What does "uncertainty of the Pareto front" mean?

[SRFU-NN]

After our meeting last week, I have done some thinking about the concept of "uncertainty of the Pareto front". I can see three things that could be meant:

1. The points that is X% likely to be behind the Pareto front (are dominated by some other point), for different values of X. This would truly be the uncertainty of the Pareto front, but it is not easy to calculate. Just thinking about what the probability looks like gets murky. I can only see Monte Carlo being useful, that is, drawing many full example functions for all objectives, making the Pareto front for each, and using that to tell the probability that a point is dominated. I don't think this is feasible, but I could be wrong. It would also break the ability to enquire for the parameters for a certain point.
2. Uncertainty of the points that are on the Pareto front. This is not the uncertainty of the Pareto front, but it is perhaps easier to understand. It is uncertainty in the objectives, which users know about.
3. The Pareto front made at different "probability of improvements" levels. At the moment, we make the Pareto front on the mean value for given parameters. We could also do that for e.g. the mean value - std.dev., and the mean value + std.dev. I can't see an obvious interpretation of these fronts, though.

If we are to implement visualisations of the Pareto front, we should first agree on what we actually mean with that. If you feel like it, we could discuss it at a meeting at some point.

[SRFU-NN]

To expand a bit about what I mean with "Just thinking about what the probability looks like gets murky", let's say that we have a set of objectives O, and a space S. Then X is dominated if there exists a point Y in S, such that for all o in O, o(Y)<=o(X), and there exist and o in O such that o(Y)<o(X). But how on earth you would calculate that probability is beyond me. You would somehow have to include all of the covariances between all of the points in S, which just gets weird.

[dk-teknologisk-mon]

Some loosely structured thoughts on this:

While we speak of "the Pareto front" as an object we will have access to, this is not really the case. In the code at the moment we estimate a finite (and frankly low, n = 40) number of points on the front. We will never know the exact location of it. With the way we estimate these points right now, we also don't get the same answer every time. In the plot below I have overlaid two successive calls of plot_Pareto_bokeh on the output from our example folder, where we are well covered by data.

I suspect that in real use cases, these estimates will risk being very different in successive calls of the plot, but this is hidden away and non-expert uses will just take it at face value - just like they used to do with the single-factor plots of plot_objective before we forced uncertainty into it.

I think the consequence of the above is that something like your point 1 is what I would prefer. Basically, I'd like to know what the distribution of Pareto fronts look like, when we sample from our posterior model of each objective. When our models are very uncertain in the beginning, this should result in a similarly wide distribution of possible fronts in objective space. When are models become good, the distribution of Pareto fronts should similarly become narrow. When data coverage is bad, it is also relatively easy to see that the output of the simulated front is pretty non-sensical:

[SRFU-NN]

To make my issue with the uncertainty on the Pareto front more concrete: I would like the ability to enquire about a point on the Pareto front that has one of the objectives within an interval. So "Can I get a recipe for brownies that has the best possible quality for their price, given that the price is between 2 and 3 kr per brownie?". Can we think of a way to do that that is consistent with concept 1 for uncertainty of the Pareto front?