Question about the influence of prior choice on Bayes factor

[zhengchencai]

Question and context
Hello,

Thank you very much for developing this amazing tool. May I ask a question regarding Bayes factor? in this tutorial https://easystats.github.io/bayestestR/articles/bayes_factors.html, you explained very well the calculation of BF, however, do you think it will be influenced by the prior choice? Imagine I now have a much wider prior than the one shown in your figure and also imagine our data is informative. Then using a wider prior would not influence too much the posterior, but odd for prior vs null will be influenced a lot since prior is now wider. Therefore the final BF will decrease. How do we avoid this? Thanks a lot.

[mattansb]

If you have very wide priors, it seems unlikely you would want to compute a Bayes factor against the null.

BF is essentially a bet - you are betting the data will on average fit your prior compared to the null. But a very wide prior means that are betting on a wide verity of values, and so you will certainly be more wrong than right (compared to the null).

---

Example:
I (the null) say that the value is 0.
You say: No, it is not 0 - it is somewhere between -100 and +100.

We collect data and find that in the range of -100 and +100, the data is relatively (to my 0) less compatible with most of that range. So on average, your "bet" was worst than mine.

With Bayes factors: More specific and nuanced priors yield higher "rewards". Non specific priors will always lose.

---

So how to avoid this?

Avoid the bet alltoghether - don't use BFs if you don't have an informative prior.

Or... use BFs, but not against a point null. You can compare "the coef is positive" to "the coef is negative":

bayesfactor_parameters(posterior, prior, null = c(-Inf, 0))

However, note that these are priors you are comparing - so changing your prior after seeing the data makes no sense (and is overfitting).

[zhengchencai]

@mattansb Thanks for your answer, I agree with what you explained here. Then can we safely use BF as the effect size and compare it across studies? Do you think it would be hard to evaluate how stable it is when different priors are used? Let's say all studies use kind of informative or weakly informative priors, would BF=3.1 from one study differs a lot comparing to BF=4 from the other study? Maybe the solution is to use the same prior if the goal is to compare effect size from different studies? I understand this has been debatable, so please feel free to ignore my questions :)

[bwiernik]

BF is not an effect size. It isn't any more comparable across studies than a p value