RAG filter in code uses external judge API, but paper says solver itself does verification

[528why]

Hi authors, thanks for the great work!

In the paper, the RAG filter verification is described as using the solver itself to answer with collected RAG documents (no search tools), to verify proposer questions:

“...we collect all the search results in the proposer’s trajectory as the RAG documents, and let the solver answer without using search tools. If the proposer’s question is correct... the solver should already have sufficient information to correctly predict the answer...”

However, in the released code, the RAG filter seems to call an external LLM-as-a-judge API, rather than the solver model:

• quarl/utils/problem_extraction.py initializes llm_judge via get_global_judge(...) with QUARK_BASE_URL / QUARK_MODEL.
• When use_rag_filter is enabled, _validate_with_external_llm(...) is called, which uses llm_judge.model_based_answer(...) and llm_judge.model_based_match(...).

This appears to be a different model/service than the solver (which is the actor model used in rollout).

1. Is it actually intended to use a different model (external judge) for the RAG filter, or should it use the solver itself as described in the paper? If the solver is intended, is the current code path incorrect or incomplete?
2. From your experience, which works better in practice: using an external judge model vs. using the solver itself for RAG verification?
3. In my experiments, using a stronger external model increases the RAG filter pass rate in early training, but it may also introduce harder questions that the solver later struggles to solve. Is this expected, and do you recommend any mitigation strategies?

[Necolizer]

Hi @528why , thank you for the careful reading and for raising these thoughtful questions.

Conceptually, the most faithful implementation is indeed to reuse the actor model used in rollout to answer using the collected RAG documents only, as described in the paper. However, in practice this is quite expensive and brittle: the solver policy is changing during training, and calling it again for RAG verification significantly hurts sampling efficiency and robustness.

For this reason, our code implements the RAG filter as a model-based rejection sampler, which can flexibly use the base model or another fixed model as a RAG solver. In our experiments (e.g., Qwen2.5-32B), the RAG filter also uses Qwen2.5-32B, but it is not trained and only serves to filter out invalid questions. A stronger RAG solver will accept more valid questions per batch, while a weaker one will accept fewer (typically a subset of those accepted by the stronger model). The latter case simply requires more proposer sampling rounds to fill a batch with valid questions, which reduces sampling efficiency.

Using a stronger external judge typically increases the early pass rate, which is expected. It may introduce a curriculum mismatch. In practice, keeping the judge comparable to (but not drastically stronger than) the base model can be a mitigation strategy for this problem.

Hope this clarifies.