How to ignore failed toy fit AssertionError

[zjkjsd]

Aloha experts,

I am fitting toys by

fit_results = np.asarray(
        [pyhf.infer.mle.fit(toy, model,return_uncertainties=True) for toy in toys]
    )

However, this only works if all fits are successful. If 1 toy fit fails, the whole loop will terminate with an AssertionError. I also tried to write the list comprehension with a try-except structure but it still give the same situation. Could you give me a hint to ignore failed fits? Thank you very much! Sorry if this has been discussed somewhere else, if so, I would appreciate it if you could direct me to the place!

def fit_toys(toy):
    try:
        return pyhf.infer.mle.fit(toy, model,return_uncertainties=True)
    except AssertionError:
        pass

[alexander-held]

What exactly doesn't work with the second approach?

[zjkjsd]

Thanks! I did:

fit_results = np.asarray(
        [fit_toys(toy) for toy in toys]
    )

The LLVM ERROR shows up when I use JAX backend, it disappears with numpy backend. But both backends give the AssertionError. Edit: Actually this LLVM ERROR appears also when I use multiprocessing (not now), this might be the cause, but probably like you said, it is an independent problem.

 p = Pool(None, maxtasksperchild=1)
fit_results = np.asarray(
     p.map(fit_toys, toys)
     )

[alexander-held]

This should ignore failed fits just fine. If it doesn't, something else is going wrong. Here's the structure you're using:

import numpy as np

def fit(x):
    if x > 0.5:
        assert False
    else:
        return {"fit": "success"}


def fit_a_toy_and_catch_exception(x):
    try:
        return fit(x)
    except AssertionError:
        return {"fit": "failed"}


toys = np.random.random(10)

results = [fit_a_toy_and_catch_exception(toy) for toy in toys]
print(results)

which does its job. Are you not catching the AssertionError successfully while using this structure?

[zjkjsd]

I can catch the AssertionError with your code example successfully. I am wondering if pyhf.infer.mle.fit(toy, model,return_uncertainties=True) already has a try-except AssertionError inside, so it would encounter the except AssertionError 2 times?

[alexander-held]

Ah I see what's probably going wrong here, indeed pyhf catches that AssertionError and turns it into something else like a pyhf.exceptions.FailedMinimization. You need to catch the latter (or catch anything via a blank except: though that is usually not a great idea). You should see something like the following:

AssertionError

During handling of the above exception, another exception occurred:
[...]
pyhf.exceptions.FailedMinimization: Inequality constraints incompatible

Catch the last assertion in the output you see.