GBM prediction confidence

[gaop123]

Hi, is it possible to add an option for getting the confidence of a prediction of a GBM?
To know how much the prediction "can be trusted"
How can i do it by my self?
Thanks!

[mdabros]

Hi @gaop123 ,

If you are creating a classification model, you can get the confidence of the GBM by using the PredictProbability method on the model:

/// Single new data point 
var observation = new double[] { 1, 2, 200, 1, 3.4 };
var probabilityPrediction = model.PredictProbability(observation);

// This is a dictionary from target value to probability/confidence
var probabilities = probabilityPrediction.Probabilities;

// targetValue of the predicted class
var prediction = probabilityPrediction.Prediction;

// This will give the probability/confidence of the predicted class.
var predictionProbability = probabilities[prediction].

If you need the probability/confidence for the other classes, you can get the probability by parsing their target values to the probabilities dictionary.

Best regards
Mads

[gaop123]

Hello, Thanks for the quick response, By confidence i meant for a way to know the probability that the input is actually belongs to one of the labels, Because the probability that the gbm model returns allways sums to one there is no way to know if the label belongs to none of the labels, Thanks! בתאריך יום ב׳, 30 ביולי 2018, 8:53, מאת Mads Dabros ‏< notifications@github.com>:

…

Hi @gaop123 <https://github.com/gaop123> , If you are creating a classification model, you can get the confidence of the GBM by using the PredictProbability method on the model: /// Single new data point var observation = new double[] { 1, 2, 200, 1, 3.4 };var probabilityPrediction = model.PredictProbability(observation); // This is a dictionary from target value to probability/confidencevar probabilities = probabilityPrediction.Probabilities; // targetValue of the predicted classvar prediction = probabilityPrediction.Prediction; // This will give the probability/confidence of the predicted class.var predictionProbability = probabilities[prediction]. If you need the probability/confidence for the other classes, you can get the probability by parsing their target values to the probabilities dictionary. Best regards Mads — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#83 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANFWWZ9mAmTMQxHyepCh56RvpFSE9LuKks5uLp9dgaJpZM4VlMkA> .

[mdabros]

@gaop123 Ahh yes, that is something different indeed. Currently, there are no build in methods for dealing with classes unknown at training time.

There are a few ways you can handle it, for instance add an anomaly detector in front of your classifier. The anomaly detectors purpose would be to flag all observations which are different from what the classifier was originally trained on, separate these out, and only parse on the observations that seem to belong to the original data distribution.
This is by no means a perfect method though, and it will have some draw backs, since the anomaly detector won't be 100% correct. Sometimes you will still get unknown classes sent to the classifier, and sometime the anomaly detector will filter out known classes.

More consideration can be found here: what-image-classifiers-can-do-about-unknown-objects