About the self-preference bias of judge model.

[JiayiFu]

When employing Claude 4.5 Sonnet as the judge in this benchmark, is there a risk that self-preference bias might lead to an overestimation of its own scores?

[tupeiyi-tpy]

Thank you for raising this important question! We've carefully considered the potential self-preference bias when using Claude as a judge and have validated our approach from multiple angles. First, when we produced the initial small batch of questions, we conducted comparative experiments using Kimi as an alternative judge and found that the human-machine agreement rates were 79% and 80% respectively, showing highly consistent results, which led us to choose Claude for its slightly higher consistency. Second, we ran experiments on another independently generated set of test questions, also using Claude as the judge, but in that dataset Claude's scores were not particularly high, demonstrating that the scoring results primarily depend on the rubric and actual performance rather than the identity of the judge model. Additionally, Claude doesn't know which model it's evaluating during the judging process, implementing a blind evaluation design. More importantly, we focus on scores across individual dimensions rather than just the total score, and we've observed that Claude scores higher in some dimensions but lower in others, showing a reasonable distribution pattern. Based on these comprehensive validations, we believe the benchmark evaluation is fair and objective.