Feature: evaluation granularity

[agataben]

Now it is possible to choose the granularity of the evaluation setting the parameter granularity in the function evaluate(). There are 3 possible choices:

1. granularity = 'variable': the detected anomalies of a specific type are counted for each variable of the series (same for false positives). For each series of the dataset, the counts are then divided for a normalization factor, given by the product of the number of variables in the series by its length.
   Suppose you have series of 2880 data points and 2 variables (temperature and humidity) like this:

Timestamp	Temperature	Humidity	Anomaly Label
1973-05-07 11:02:00+00:00	24.761107302835810	50.63704719243785	'spike_uv'
1973-05-07 11:17:00+00:00	24.868377982941322	50.350992045489804	None
1973-05-07 11:32:00+00:00	24.944096137309916	50.14907696717356	None
...	...	...	...
1973-05-07 11:47:00+00:00	15.987937529180115	59.03216658885302	'step_uv'
1973-05-07 12:02:00+00:00	24.999714422981285	50.000761538716574	'step_uv'
1973-05-07 12:17:00+00:00	24.979376388246145	50.05499629801029	'step_uv'
1973-05-07 12:32:00+00:00	24.927010515561776	50.194638625168594	'spike_uv'
...	...	...	...

there are 5 anomalies, 2 of type 'spike_uv' for the temperature and 3 of type 'step_uv' for the humidity. The model sees both the anomalies on the temperature but only 1 anomaly on the humidity and gives 2 false positives, one on temperature and one on humidity at the same timestamp.

Timestamp	Anomaly Label	Temperature Anomaly	Humidity Anomaly
1973-05-07 11:02:00+00:00	'spike_uv'	1	0
1973-05-07 11:17:00+00:00	None	1	0
1973-05-07 11:32:00+00:00	None	0	1
...	...	...	...
1973-05-07 11:47:00+00:00	'step_uv'	0	1
1973-05-07 12:02:00+00:00	'step_uv'	0	0
1973-05-07 12:17:00+00:00	'step_uv'	0	0
1973-05-07 12:32:00+00:00	'spike_uv'	1	0
...	...	...	...

The evaluation of the model on this series will be the following dictionary:

series_evaluation = { 'spike_uv': 0.000347,  'step_uv': 0.00017, 'false_positives': 0.000347}

Of course, this granularity can be used on models that assign anomaly flags to each variable of the series
2) granularity = 'data_point': the detected anomalies of a specific type are counted for each data point of the series. In this case the normalization factor is only the length of the series. For the same example as before, the evaluation result on the series with point granularity will be:

series_evaluation = { 'spike_uv': 0.000694,  'step_uv': 0.000347,  'false_positives': 0.000694}

The "series_evaluation" dictionary is created for each series of the dataset. To obtain the evaluation on the dataset, values corresponding to the same key are summed.
This is done for each model.
3) granularity=series: having only one type of anomaly per series is necessary to use this granularity. The anomalous series of the dataset detected as anomalous raise the counts of the type of anomaly. Not anomalous series detected as anomalous raise the counts of false positives.
Suppose we have a dataset with 3 normal series and 4 anomalous series with spike_uv, step_uv, pattern_uv and clouds anomalies. The model detects spike_uv and clouds anomalies and gives 1 false positives. The result of the evaluation is:

evaluation: { 'spike_uv': 0.143, 'clouds': 0.143, 'step_uv': 0.000, 'fase_positives': 0.143
}

Values are normalized to the number of series inside the dataset