Relative Difference of Mean not detecting large change

[johnaohara]

A change detection variable defined with the following relative difference of means

is not detecting large changes after the first 2 changes have been detected;

[johnaohara]

the reason it was not fired was due to the number of proceeding datapoints in the configuration is set to 5. The relative difference of means requires 5 data points from the last change point. so in this case the large changes were not analysed as there were not enough datapoints from the previous change.

[jhutar]

Hello John. What would be a fix here? Given our specific data, would you have a recommendation on change detection config?

[johnaohara]

@jhutar if your case, you can reduce the "Minimal Number of preceeding datapoints" to 1, although I would keep the threshold quite high to stop noisy alerts.

I think we need to improve the docs on threshold configuration and the implications

[naftalysh]

Hi @johnaohara
Is it a bug or a feature and if it's a feature could it updated to have the preceding sliding window not starting from a change notification datapoint but a regular inclusive sliding window?

[johnaohara]

@naftalysh it is a feature.

As "Relative Difference of means" is implemented atm, it assumes that we have a stable period of "Minimal Number of preceding datapoints" before change detection starts. Ofc this means that once a change is detected, it is assumed that that state is consistent for the next window size. This assumption does not hold where there are multiple, sequential changes occur. However idk the probability of this occurring in reality.

historically, we have set 2 change point detection conditions, one with a sliding window of say 5, with a threshold of 10-20%, and another with a window of 1, with a threshold of 30-40%. The second would capture the changing timeseries data with subsequent data points, although it would require 2 more datapoints after a change before a change is detected.

With the current implementation, if we have 3 stable periods with the number of samples greater than "Minimal Number of preceeding datapoints", change detection would fire at these points;

If we modify the algorithm to use a "regular inclusive sliding window", using the same dataset as before, we would trigger the following change detection events.

This increases the the number of false positives, and introduces noise.

The issue with the current implementation is if we have a time series dataset where there a number of changes in quick succession, these are the events that would fire with a "Minimal Number of preceding datapoints" of 5 and a threshold of 20%;

We see a lot of false negatives, i.e. there are changes that are not detected. However, I am not sure of the probability of seeing a time series of performance results of this form

We can currently mitigate the false negatives, but setting up another change detection config as previously described, removing the false positives with a "regular sliding window" would be more difficult .

Maybe we should be implementing another change detection algorithm other than "relative diffence of mean", that reduces false positives and false negatives

[naftalysh]

Hi @johnaohara
Thanks for the answer
I shall read it carefully