A standardized (i.e., absolute) index of uncertainty?

[DominiqueMakowski]

Holidays got me thinking, it could be interesting to have a metric of uncertainty (i.e., of the width of the posterior) that is absolute (i.e., independent of scale) that could be used to facilitate interpretation of the estimation of uncertainty. Indeed, currently, guidelines suggest to just report the CI but its width is in actuality rarely commented (aide from checking it contains the null). Which is interesting as people have developed interpretation guides that they apply to the point-estimates (effect sizes; small, medium etc.), the existence/significance indices, but not for the uncertainty... Why? And what would be suitable?

Of course, an index of dispersion, such as the SD immediately came to my mind. It would have to be obtained from the standardized posterior (hence it'll have to be implemented in parameters alongside a standardize_posterior() helper). The MAD would possibly be better, but might still be limited in the case of skewed distributions. Thus, maybe the difference between the upper and lower CI bounds of the standardized posterior might be used? What would make more sense, HDI or ETI? How does such an index of uncertainty relate to other indices or main sources of variance?

[mattansb]

Interesting! I like the idea of a diff(CI)/MAD, only problem is that the width of the CI is highly correlated to the MAD, so this ratio itself might be basically fixed?

Testing:

library(rstanarm)
library(bayestestR)


fit <- stan_glm(mpg ~ ., data = mtcars) 

fit2 <- stan_glm(switch ~ ., data = wells)

unlist(lapply(list(fit, fit2), function(x){
  x <- describe_posterior(x, dispersion = TRUE)
  (x$CI_high - x$CI_low) / x$MAD
}))
#> Possible multicollinearity between wt and disp (r = 0.76). This might lead to inappropriate results. See 'Details' in '?rope'.
#>  [1] 3.123609 3.262108 3.131296 3.276669 3.248299 3.258730 3.267935
#>  [8] 3.124618 3.356547 3.279599 3.322555 3.218256 3.118310 3.238458
#> [15] 2.954221 3.138057

^{Created on 2019-09-07 by the reprex package (v0.3.0)}

I think that MAD or diff(CI) from a standardized posterior might work better :)

[DominiqueMakowski]

Some quick plots for an index of uncertainty defined as the difference between higher and lower 89% HDI (based on the data used for the paper)

code

```r library(tidyverse) library(rstanarm) library(bayestestR) library(estimate) library(see)

df <- read.csv("https://raw.github.com/easystats/easystats/master/publications/makowski_2019_bayesian/data/data.csv")
df <- df %>%
mutate(Uncertainty = CI_high_89_hdi - CI_low_89_hdi,
true_effect = as.factor(true_effect))

df %>%
mutate(sample_size = as.factor(sample_size)) %>%
ggplot(aes(x = sample_size, y = Uncertainty, fill = sample_size)) +
geom_violin() +
scale_fill_viridis_d(guide = FALSE) +
facet_grid(~outcome_type) +
theme_modern()

df %>%
mutate(temp = as.factor(cut(error, 6, labels = FALSE))) %>%
group_by(temp) %>%
mutate(noise = as.factor(round(mean(error)))) %>%
ungroup() %>%
ggplot(aes(x = noise, y = Uncertainty, fill = noise)) +
geom_violin() +
scale_fill_viridis_d(guide = FALSE) +
facet_grid(~outcome_type) +
theme_modern()

model <- lm(Uncertainty ~ true_effect / outcome_type / poly(sample_size, 2) * error, data = df)
summary(model)

model %>%
estimate::visualisation_matrix(target = c("error = 0:6", "sample_size = 0:100", "true_effect = as.factor(1)", "outcome_type = 'linear'")) %>%
estimate::estimate_link(model) %>%
ggplot(aes(x = sample_size, y = Predicted, color = error, group = error)) +
geom_line(size = 1) +
scale_color_viridis_c() +
theme_modern()

</details>