pre-processed training data

[minamonaa]

I have two questions about the pre-processed training data of NQ data.
How is it possible for 'has_gold_answer' to be False when 'em' is 1 and 'f1' is 1.0?
What criteria were used to select 'positive_ctxs'? In QA tasks, it is mentioned that the context with the highest EM score was chosen, but how were 'positive_ctxs' set when there were multiple sentences with an EM of 1?

[carriex]

Hi there,

has_gold_answer denotes whether the retrieved documents contains the gold answer, while EM and F1 measures whether the model outputs the correct answer. It is possible that model outputs the correct answer while it is not in the retrieved documents.

When there are multiple sentences with EM equals to 1, we select the one with the highest P(gold_answer | sentence) (i.e. the sentence which leads to the highest probability of the gold answer when prepended).

Hope that helps!