Details of Evaluation Set

[CooFlow]

Hi, I notice the data preprocessing code in "verl/dataset/bird.py":

# 过滤test_dataset中prompt长度超过8000的项
test_dataset = test_dataset.filter(lambda x: len(x["instruction"]) <= 10000)
train_dataset = train_dataset.filter(lambda x: len(x["instruction"]) <= 11000)

length for train.parquet 8454
length for test.parquet 1405

Does this mean the experiment scores were completed on a subset of BIRD-DEV, after removing some of the very long ones?

[facdbe-kt]

Thank you for your question!

The code you mentioned in "verl/dataset/bird.py" is used for data processing in the reinforcement learning (RL) part of VERL. This filtered dataset is only used for training and evaluation during the reinforcement learning process. Due to GPU memory constraints, we need to filter the data by length during the GRPO process to prevent CUDA out of memory errors.

However, for the actual model evaluation and the experimental scores reported in our results, we conducted the evaluation on the complete test dataset without any length filtering. The filtering is only applied during the RL training phase to ensure the training process can run smoothly within memory limitations.

So to directly answer your question: No, the experiment scores were completed on the full BIRD-DEV dataset, not on a subset. The length filtering you observed is specifically for the RL training process, not for the final evaluation.