deep research rubric prompts -- rigor

[MikeACedric]

@jd-coderepos , the prompt for the rubric "Statistical Sophistication" is below:

from ..base import Rubric

statistical_sophistication_prompt = """<Context>
Scientific question answering and synthesis often require more than listing findings: high-quality scientific writing demonstrates rigorous reasoning using quantitative evidence. This is commonly expressed through statistical sophistication, where the text presents and interprets statistics like variance, standard deviation, standard error and evaluates the robustness and reproducibility of the results.

The response may be a single paragraph or a long-form report with multiple sections. There are no strict requirements on length or formatting; statistical sophistication should be evaluated independently of presentation style.

This rubric focuses exclusively on the presence and quality of statistical reasoning within the provided text, emphasizing the appropriate use, interpretation, and evaluation of statistical analysis rather than mere reporting of results. Other aspects of scientific quality (such as mechanistic understanding, factual accuracy, or completeness) are intentionally outside its scope and are assessed by separate evaluation criteria.
</Context>

<Role>
You are tasked as a scientific writing quality evaluator.
</Role>

<Task-Description>
A user will provide you with:
1) a research question, and
2) a written response intended to address that question.

You must evaluate the response using the evaluation characteristic below. Focus on whether the response demonstrates statistical sophistication by clearly applying, interpreting, and reasoning with statistical analysis, rather than merely reporting descriptive results. Your judgment should be based solely on the provided question and response.
</Task-Description>

<Evaluation-Characteristics>
StatisticalSophistication: Does the response demonstrate statistical sophistication by appropriately applying, interpreting, and reasoning with statistical analysis (e.g., variance, standard deviation, standard error), rather than only reporting descriptive results or outcomes?
</Evaluation-Characteristics>

<Domain-Vocabulary-Examples>
Below are domain-specific terms and phrases that often signal statistical sophistication. They are examples only: their presence is not required, and their presence alone is not sufficient for a high score.

{STATISTICAL_SOPHISTICATION_VOCAB}
</Domain-Vocabulary-Examples>

<Rating-Scale>
For the characteristic above, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below.

StatisticalSophistication
Rating 1. Very bad: The response is purely descriptive, reporting results or outcomes with no meaningful use, interpretation, or reasoning of statistical analysis.
Rating 2. Bad: The response contains occasional statistical terms or references, but application or interpretation is superficial, generic, or weakly connected to the research question.
Rating 3. Moderate: The response demonstrates some statistical reasoning with partial detail, but key analysis, interpretations, or considerations of variability and assumptions are missing, unclear, or inconsistently applied.
Rating 4. Good: The response applies statistical analysis appropriately and interprets results with reasonable clarity, minor fluctuations or imprecision may remain.
Rating 5. Very good: The response demonstrates thorough statistical sophistication, correctly applying, interpreting, and reasoning with multiple relevant analysis, clearly addressing assumptions and tightly connecting statistical evidence to the research question.

</Rating-Scale>

<Response-Format>
Rate the quality from 1 (very bad) to 5 (very good). Provide a short rationale that highlights specific aspects of the response demonstrating the appropriate use, interpretation, and reasoning of statistical analysis, or the absence thereof, in addressing the research question.

Return your response in JSON format:
{
  "StatisticalSophistication": {"rating": "", "rationale": ""}
}
</Response-Format>

<Example-Responses>

{EXAMPLE_RESPONSES}

</Example-Responses>

<Note>
Your evaluation must be based solely on the provided research question and response. Do not reward verbosity and descriptive results; reward appropriate use, interpretation, and reasoning of statistical analysis, relevance to the question, and clarity in connecting statistical evidence to conclusions. This rubric does not assess mechanistic understanding, factual accuracy, or completeness.
</Note>"""
class StatisticalSophistication(Rubric):
    name: str = "StatisticalSophistication"
    system_prompt_template: str = statistical_sophistication_prompt

Vocab to be used for "Statistical Sophistication":
"stats_terms" for both ecology and nlp

[jd-coderepos]

first comment, I would not call this statistical sophistication. This sounds unnecessarily complicated. :)

Also for the vocab, it is

"rigor_stats": [
"p-value", "p<", "p >", "significant", "confidence interval", "ci",
"t-test", "anova", "regression", "bootstrap", "cross-validation",
"held-out", "standard deviation", "std", "mean", "median"
],
"stats_terms": [
"p-value", "confidence interval", "t-test", "anova", "regression",
"effect size", "variance", "standard deviation", "standard error", "r-squared"
],

for NLP. link Isn't it?

And

"stats_terms": [
"mean", "median", "variance", "standard deviation", "standard error",
"confidence interval", "ci", "p-value", "significant", "regression",
"anova", "t-test", "chi-square", "effect size", "meta-analysis",
"model comparison", "r-squared"
],

for Ecology which is fine.

I will suggest some changes to the vocabs as well below.

[jd-coderepos]

I would suggest to use this instead. So replace "Statistical Sophistication" with QuantitativeEvidenceAndUncertainty

from ..base import Rubric

quantitative_evidence_and_uncertainty_prompt = """<Context>
Scientific question answering and synthesis often require more than listing findings: high-quality scientific writing demonstrates rigorous reasoning using quantitative evidence and an appropriate treatment of uncertainty. In synthesis settings (e.g., reports that summarize multiple papers), this is commonly expressed through careful use and interpretation of source-reported quantitative results (e.g., effect sizes, confidence intervals, variability measures), and through explicit reasoning about robustness, heterogeneity, and limitations of the evidence base.

Importantly, not every research question or evidence base requires (or supports) statistical detail. Some questions are primarily conceptual, mechanistic, methodological, or qualitative; in such cases, it can be rigorous to state that quantitative estimates are unavailable, incomparable, or not applicable—and to explain why (e.g., heterogeneous outcomes/metrics, qualitative designs, sparse data).

The response may be a single paragraph or a long-form report with multiple sections. There are no strict requirements on length or formatting; quantitative evidence and uncertainty handling should be evaluated independently of presentation style.

This rubric focuses exclusively on the presence and quality of quantitative evidence handling and uncertainty reasoning within the provided text, emphasizing appropriate use, interpretation, and limitation-aware reasoning rather than mere mention of statistical terms or ungrounded numerical claims. Other aspects of scientific quality (such as mechanistic understanding, factual accuracy, evidential grounding, or completeness) are intentionally outside its scope and are assessed by separate evaluation criteria.
</Context>

<Role>
You are tasked as a scientific writing quality evaluator.
</Role>

<Task-Description>
A user will provide you with:
1) a research question, and
2) a written response intended to address that question.

You must evaluate the response using the evaluation characteristic below. Focus on whether the response demonstrates rigorous quantitative reasoning and uncertainty awareness appropriate to the research question and the type of synthesis, rather than merely reporting outcomes or sprinkling statistical jargon. Your judgment should be based solely on the provided question and response.
</Task-Description>

<Evaluation-Characteristics>
QuantitativeEvidenceAndUncertainty: Does the response appropriately use and interpret quantitative evidence and uncertainty (e.g., effect sizes, confidence intervals, variability, robustness, heterogeneity) in a way that is relevant to the research question? If such quantitative treatment is not needed or not supported by the evidence base, does the response explicitly and reasonably justify why?
</Evaluation-Characteristics>

<Domain-Vocabulary-Examples>
Below are domain-specific terms and phrases that often signal quantitative evidence handling and uncertainty reasoning. They are examples only: their presence is not required, and their presence alone is not sufficient for a high score.

{QUANT_UNCERTAINTY_VOCAB}
</Domain-Vocabulary-Examples>

<Rating-Scale>
For the characteristic above, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below.

QuantitativeEvidenceAndUncertainty
Rating 1. Very bad: The response is purely descriptive, reporting claims or outcomes with no meaningful quantitative evidence use or uncertainty reasoning, even when the question clearly calls for it; or it makes unqualified quantitative-sounding claims without interpretation.
Rating 2. Bad: The response contains occasional quantitative/statistical terms or numbers, but interpretation is superficial, generic, misapplied, or weakly connected to the research question; uncertainty/robustness is largely ignored when relevant.
Rating 3. Moderate: The response demonstrates some relevant quantitative reasoning and/or acknowledges uncertainty, but key interpretations, caveats (e.g., variability, assumptions, comparability), or cross-study considerations (e.g., heterogeneity) are missing, unclear, or inconsistently applied. If quantitative treatment is not needed, the response may gesture at this but without a clear justification.
Rating 4. Good: The response uses and interprets quantitative evidence appropriately where relevant and connects it to the research question; it discusses uncertainty/robustness with reasonable clarity (e.g., variability, confidence, limitations, heterogeneity) and avoids overclaiming. If quantitative treatment is not needed or not supported, it states this clearly and provides a reasonable explanation.
Rating 5. Very good: The response demonstrates strong quantitative rigor and uncertainty awareness tightly aligned with the research question, correctly interpreting multiple relevant quantitative indicators (e.g., effect magnitude and uncertainty), explicitly addressing key assumptions/limitations and cross-study heterogeneity/robustness where applicable, and clearly distinguishing what is supported by quantitative evidence from what is not. If quantitative treatment is not needed or not supported, it provides a precise, well-reasoned justification and adjusts conclusions accordingly.

</Rating-Scale>

<Response-Format>
Rate the quality from 1 (very bad) to 5 (very good). Provide a short rationale that points to specific parts of the response demonstrating appropriate (or inappropriate/absent) use and interpretation of quantitative evidence and uncertainty, or a clear justification for why such treatment is not needed or not supported.

Return your response in JSON format:
{
  "QuantitativeEvidenceAndUncertainty": {"rating": "", "rationale": ""}
}
</Response-Format>

<Example-Responses>

{EXAMPLE_RESPONSES}

</Example-Responses>

<Note>
Your evaluation must be based solely on the provided research question and response. Do not reward verbosity or statistical jargon by itself; reward relevant quantitative interpretation, uncertainty/robustness awareness, appropriate caveats, and clarity in connecting quantitative evidence to conclusions. This rubric does not assess mechanistic understanding, factual correctness, evidential grounding, or completeness.
</Note>"""

class QuantitativeEvidenceAndUncertainty(Rubric):
    name: str = "QuantitativeEvidenceAndUncertainty"
    system_prompt_template: str = quantitative_evidence_and_uncertainty_prompt

[jd-coderepos]

And use this vocabulary instead. Ignore what is here

{
  "quant_uncertainty_vocab_ecology": [
    "effect size", "odds ratio", "risk ratio", "hazard ratio",
    "confidence interval", "CI", "credible interval",
    "standard error", "SE", "standard deviation", "SD", "variance",
    "p-value", "sample size", "n =",
    "regression", "mixed-effects", "random effects", "GLMM",
    "AIC", "model comparison", "R-squared",
    "meta-analysis", "heterogeneity", "I-squared", "sensitivity analysis"
  ]
}

{
  "quant_uncertainty_vocab_nlp": [
    "mean", "median", "standard deviation", "std", "standard error", "SE",
    "confidence interval", "CI", "bootstrap",
    "p-value", "paired t-test", "McNemar", "Wilcoxon",
    "cross-validation", "held-out", "train/dev/test",
    "across seeds", "random seed", "mean±std",
    "ablation", "robustness", "OOD"
  ]
}

[jd-coderepos]

Also under the Rigor category, I think one rubric is missing which is Uncertainty. See thesis, page 49. Can you come up with a prompt for this?

[MikeACedric]

Hi @jd-coderepos , I do seem to have missed the prompt for the rubric "Explicit Uncertainty". Can you please review this one for me:

explicit_uncertainty_prompt =  """<Context>
Scientific question answering and synthesis often require more than listing findings: high-quality scientific writing communicates to the reader when conclusions or underlying knowledge remain uncertain, unclear, or unknown.

The response may be a single paragraph or a long-form report with multiple sections. There are no strict requirements on length or formatting; explicit uncertainty should be evaluated independently of presentation style.

This rubric focuses exclusively on whether the provided text explicitly identifies knowledge as uncertain, unclear, or unknown, rather than treating all statements as resolved. Other aspects of scientific quality (such as factual completeness, causal reasoning, or relevancy) are intentionally outside its scope and are assessed by separate evaluation criteria.
</Context>

<Role>
You are tasked as a scientific writing quality evaluator.
</Role>

<Task-Description>
A user will provide you with:
1) a research question, and
2) a written response intended to address that question.

You must evaluate the response using the evaluation characteristic below. Focus on the explicit indication of uncertainty, unclear points, or unknown aspects, rather than on fully resolved statements. Your judgment should be based solely on the provided question and response.
</Task-Description>

<Evaluation-Characteristics>
ExplicitUncertainty: Does the response clearly indicate what is uncertain, unclear, or unknown in relation to the research question, rather than presenting all findings as settled or certain?
</Evaluation-Characteristics>

<Domain-Vocabulary-Examples>
Below are domain-specific terms and phrases that often signal uncertainty discussion. They are examples only: their presence is not required, and their presence alone is not sufficient for a high score.

{UNCERTAINTY_VOCAB}
</Domain-Vocabulary-Examples>

<Rating-Scale>
For the characteristic above, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below.

Rating 1. Very bad: The response provides no indication of uncertainty, all statements are presented as settled, with no acknowledgment of what is uncertain, unclear, or unknown.
Rating 2. Bad: The response occasionally signals uncertainty, but markers of what is uncertain, unclear, or unknown are vague, or weakly connected to the research question.
Rating 3. Moderate: The response identifies some aspects that are uncertain, unclear, or unknown, but important elements are missing, inconsistently marked, or only partially explained.
Rating 4. Good: The response clearly signals multiple uncertainties, unclear points, or unknown aspects relevant to the research question, minor gaps or imprecision in the explicit marking may remain.
Rating 5. Very good: The response provides a detailed, coherent account of what is uncertain, unclear, or unknown, explicitly and consistently marking multiple aspects of unresolved knowledge and clearly distinguishing these from statements presented as certain or settled.
</Rating-Scale>

<Response-Format>
Rate the quality from 1 (very bad) to 5 (very good). Provide a short rationale that points to specific parts of the response demonstrating the presence or absence of uncertainty relevant to the research question.

Return your response in JSON format:
{
  "ExplicitUncertainty": {"rating": "", "rationale": ""}
}
</Response-Format>

<Example-Responses>

{EXAMPLE_RESPONSES}

</Example-Responses>

<Note>
Your evaluation must be based solely on the provided research question and response. Do not reward length by itself; reward clear and relevant identification of what is uncertain, unclear, or unknown. This rubric does not assess factual completeness, causal reasoning, or relevancy.
</Note>"""

class ExplicitUncertainty(Rubric):
    name: str = "Explicit Uncertainty"
    system_prompt_template: str = explicit_uncertainty_prompt

Vocab to be used for "Explicit Uncertainty":
uncertainty_terms for both ecology and nlp

[MikeACedric]

The vocab from deep-research is very limited for this rubric, Should I add more terms to it?
"uncertainty_terms": [
"uncertain", "unclear", "unknown"
]

[jd-coderepos]

Ok thanks. The prompt looks fine. Yes please try to find more terms to add to the vocab.