Clarification on which dataset to sample on when using time varying parameters

[querbesd]

Hello,

I'm using both time varying intercept as well as time varying media baseline for my MMM (trained on a train set...).

To assess prediction capability of my MMM, I compare the out-of-sample predictons with observed data.

However, I get different results depending on which data set the sampling is done. When sampling on test set only, the posterior predictive mean for test data is underestimated compared to when sampling on the full data set (train + test).

As my model includes time varying parameters, I guess the best option is to sample on the full data set to capture the trends (both for intercept and media baseline) modelled on the train set. But I'd like this to be confirmed as most notebooks shows that out of sample is implemented after sampling on test set only.

🙏

[querbesd]

After digging further into it, most of the discrepancy comes from the intercept component. When sampling on test set only, the intercept is around 20% lower than when sampling on the full dataset.

Here is the chart associated with my case:

I compared my case with this notebook - example 1

I still get a discrepancy but it's much smaller.

I don't really understand the reasons for this gap when sampling on the test set. Any clue? 🤔