expected CLs calculation

[yasaman-h]

Dear experts,
Thanks to the pyhf team for providing this package!

In the first example provided here:
https://pyhf.readthedocs.io/en/v0.7.6/#:~:text=pyhf%20tutorial.-,Hello%20World,-%EF%83%81
is there a method to determine the expected CLs (CLs_exp) only based on the signal and background, without requiring the data?

[matthewfeickert]

@yasaman-h This is covered in the Introduction to HistFactory Models section of the pyhf User Guide, but the observations are a required argument, so to get what you'd like, you would need to just set the data argument of pyhf.infer.hypotest to be the background only values.

To do this programatically you would do something like the following, given the example you linked

import pyhf
pyhf.set_backend("numpy")
model = pyhf.simplemodels.uncorrelated_background(
    signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
)

bkg_pars = model.config.suggested_init()
bkg_pars[model.config.poi_index] = 0  # set the poi value to 0
bkg_only_obs = model.expected_actualdata(bkg_pars)
data = bkg_only_obs.tolist() + model.config.auxdata

test_mu = 1.0
CLs_obs, CLs_exp = pyhf.infer.hypotest(
    test_mu, data, model, test_stat="qtilde", return_expected=True
)
print(f"Observed: {CLs_obs:.8f}, Expected: {CLs_exp:.8f}")
# Observed: 0.06600882, Expected: 0.06600882

[yasaman-h]

Hi,
The reported CLs_obs and CLs_exp correspond to what Confidence Level? 68% CL or 95% CL?

[matthewfeickert]

@yasaman-h A test statistic doesn't have a confidence level. The $(1-\alpha)\cdot100$% confidence level is something that you select during limit setting. c.f. the pyhf.infer.intervals.upper_limits.upper_limit docs and the Simple Upper Limit example in the Introduction to HistFactory Models section of the pyhf user guide.

For more on limit setting, check out the pyhf user guide section on Model-Independent Upper Limits.