Error evaluating a new model

[Felipehonorato1]

After generating the predictions file and running evaluation.py i got this message:

File "evaluation.py", line 90, in count
if (self.id2score[pro_id] > self.id2score[anti_id]):
KeyError: '6b56153532fa360d37c25e918546f571'

[sk-g]

were you able to solve this?

[Felipehonorato1]

@sk-g Not really... Just commented the missing indexes and ran it again. It worked but not properly.

[sk-g]

@Felipehonorato1 I found a work around, I noticed that this is happening (at least in my case) with --predictions-dir but when I run individually on each file with --predictions-file, it works just fine on the same predictions.

[VelikayaScarlet]

same
Evaluating pred/stereoset_m-BertForMaskedLM_c-bert-base-uncased_s-42.json...# my fine tuned model
Traceback (most recent call last):
File "/content/StereoSet/code/evaluation.py", line 193, in
parse_file(args.gold_file, prediction_file)
File "/content/StereoSet/code/evaluation.py", line 151, in parse_file
score_evaluator = ScoreEvaluator(
File "/content/StereoSet/code/evaluation.py", line 65, in init
results[split][domain] = self.evaluate(self.domain2example[split][domain])
File "/content/StereoSet/code/evaluation.py", line 76, in evaluate
counts = self.count(examples)
File "/content/StereoSet/code/evaluation.py", line 90, in count
if (self.id2score[pro_id] > self.id2score[anti_id]):
KeyError: '6b56153532fa360d37c25e918546f571'