Onboarding SimpleQA

[vidhishanair]

This PR onboards SimpleQA based on https://github.com/openai/simple-evals/blob/main/simpleqa_eval.py

[luizvalle]

nit: Would it be good to document what is the expected format of the metadata column? Just so it's easier for someone who looks at this file to know what it should look like without having to look at other parts of the code.

[luizvalle]

If x is an invalid Python literal, ast.literal_eval() will raise an exception. Do we want to handle this gracefully here or is it okay to let the program crash in this case?

[luizvalle]

If the first row has a None, there is will be an exception since None does not have a keys() attribute. But, seems like this case should not happen. Flagging just in case.

[luizvalle]

Is this column a boolean indicating whether the response was correct or a count of correct responses? May be good to clarify this in the docstring.

[luizvalle]

nit: An alternative to

str(grading_response)=="nan"

could be

pd.isna(grading_response)

I think this would usually be safer, but up to you.

[luizvalle]

nit: This instruction seems slightly misaligned with the final one:

Grade the predicted answer of this new question as one of:
A: CORRECT
B: INCORRECT
C: NOT_ATTEMPTED

Just return the letters "A", "B", or "C", with no text around it.

[luizvalle]

Same as above: The model should output A, B, or C but this instruction tells the model to output CORRECT, INCORRECT, NOT ATTEMPTED>

[luizvalle]

Should we remove the sampling before submitting?

[luizvalle]

Just for my understanding, why are these two transforms needed? Couldn't we just use SimpleQA_Metric_grade?

[luizvalle]

Also just so I can understand: Why is it needed to rename the index?

[luizvalle]

Can we add a comment explaining why the explode is not necessary in this case? Maybe the verified dataset does not have the metadata column which is being exploded?

[luizvalle]

If the grading model outputs something other than only one of A, B, or C, then this regex may result in false positives.

For example:

import re
re.search(r"(A|B|C)", "A grading response that has some of the letters B, C")

would match the article 'A' at the beginning of the sentence.

Maybe a more strict regex would do?

match = re.search(r"^(A|B|C)$", grading_response)

[luizvalle]

Do we need this original_group_by variable? Could we just use self.group_by instead?

[luizvalle]

Maybe we can pull out some of the processing functions into helpers just to make each function smaller?

Example:

import re
import pandas as pd
from eureka_ml_insights.metrics.metrics_base import CompositeMetric
from eureka_ml_insights.metrics.reports import NumericalAggregator

GRADE_REGEX = re.compile(r"^(A|B|C)$")
GRADE_DEFAULT = "C"


def _parse_grade(response):
  """Extract grade A/B/C from the model response."""
  if response is None or pd.isna(response):
      return GRADE_DEFAULT
  
  match = GRADE_REGEX.search(str(response))
  return match.group(0) if match else GRADE_DEFAULT


def _process_row(self, row):
  grade = _parse_grade(row["model_output"])
  
  return {
      "grade": grade,
      "is_correct": grade == "A",
      "is_incorrect": grade == "B",
      "is_not_attempted": grade == "C",
  }


class SimpleQA_Metric(CompositeMetric):
    """
    Composite metric for evaluating SimpleQA responses.
    """

    def __evaluate__(self, row):
        return self._process_row(row)


class SQA_CGAAggregator(NumericalAggregator):
    """
    Computes accuracy = correct / attempted,
    where attempted = correct + incorrect.
    """

    def __init__(self, correct_col, incorrect_col, not_attempted_col, output_dir, group_by=None, **kwargs):
        super().__init__(
            [correct_col, incorrect_col, not_attempted_col],
            output_dir,
            group_by=group_by,
            **kwargs,
        )
        self.correct_col = correct_col
        self.incorrect_col = incorrect_col
        self.not_attempted_col = not_attempted_col

    def _compute_accuracy(self, df):
        attempted = df[self.correct_col].sum() + df[self.incorrect_col].sum()
        if attempted == 0:
            return 0.0
        return df[self.correct_col].sum() / attempted

    def _aggregate(self, data):
        self.aggregated_result = {
            "accuracy_given_attempted": self._compute_accuracy(data)
        }

    def _aggregate_grouped(self, data):
        grouped = data.groupby(self.group_by)
        results = {}

        for name, group in grouped:
            results[name] = self._compute_accuracy(group)

        self.aggregated_result = {"accuracy_given_attempted": results}


class SQA_CGAAvgPass1Aggregator(SQA_CGAAggregator):
    """
    Computes accuracy per repeat (if present), then averages across repeats.
    """

    def _aggregate(self, data):
        # Handle pass-1 repeat grouping
        if "data_repeat_id" not in data.columns:
            return super()._aggregate(data)

        # Temporarily override grouping
        self.group_by = "data_repeat_id"
        super()._aggregate_grouped(data)

        group_results = self.aggregated_result["accuracy_given_attempted"].values()
        mean_acc = sum(group_results) / len(group_results) if group_results else 0.0
        self.aggregated_result = {"accuracy_given_attempted": mean_acc}

    def _aggregate_grouped(self, data):
        # If grouping by repeat, use the special logic above
        if self.group_by == "data_repeat_id":
            return self._aggregate(data)

        # Otherwise, compute accuracy within each external group
        grouped = data.groupby(self.group_by)
        results = {}

        for name, group in grouped:
            super()._aggregate(group)
            results[name] = self.aggregated_result["accuracy_given_attempted"]

        self.aggregated_result = {"accuracy_given_attempted": results}