Human baseline

[rbroc]

We have credits on Prolific to collect data from humans. A few outstanding questions:

• How do we sample the texts? How many texts per participant?
• How do we present the text? Show / Don't show completions? Any context on the task the text comes from?
• Do we want participants to explain their heuristics?

[yuri-bizzoni]

I would avoid giving context on where the texts came from, unless we provide the models with the similar information, as it would add a bias that we would then need to justify.
Asking the participants to explain the heuristics can be cool, but it would take, I think, a substantial time for them to write their reasonings behind their choices. If our main focus is on the textual differences between human and models, perhaps that would risk to put us a bit on a sidetrack?