From 3b74cbbe476a2bfb5a9296fa3f6bc42a5a55d7b1 Mon Sep 17 00:00:00 2001 From: Puja Trivedi Date: Fri, 6 Feb 2026 17:03:10 -0800 Subject: [PATCH 1/4] cllm integration design doc --- doc/design/cllm_integration.md | 64 ++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 doc/design/cllm_integration.md diff --git a/doc/design/cllm_integration.md b/doc/design/cllm_integration.md new file mode 100644 index 0000000..558b0c3 --- /dev/null +++ b/doc/design/cllm_integration.md @@ -0,0 +1,64 @@ +# CLLM Overview and Integration Proposal +## Overview of [CLLM (Claim LLM)](https://github.com/OpenEvalProject/cllm) +A command-line tool for extracting, evaluating, and comparing scientific claims using Large Language Models. + +CLLM implements a 4-stage workflow for scientific claim verification: + +1. **Extract Claims**: Extract atomic factual claims from a manuscript +2. **LLM Evaluation**: Group claims into results and evaluate them +3. **Peer Review Evaluation**: Group claims based on peer review commentary +4. **Compare Results**: Compare LLM and peer review evaluations + +## MVP +Ingest assertions and evidence from scientific papers into the knowledge graph using CLLM, with the following steps: +1. **Input**: Accept PDF and XML/Text inputs of scientific papers. +2. **CLLM Processing**: Run the CLLM workflow to extract claims and evaluate them using LLM review. +3. **Output Processing**: Filter the CLLM outputs and format them to map to the BICAN Assertion/Evidence model. +4. **Knowledge Graph Ingestion**: Ingest the processed outputs into the knowledge graph for downstream applications. +### Todo: +1. Add PDF Parsing Layer to extract text from PDFs and feed it into the CLLM workflow. +2. Implement output processing to filter and format CLLM outputs for knowledge graph ingestion. + +## Future Enhancements +- **Peer Review Integration**: Incorporate the peer review evaluation by working with domain-experts to annotate claims and evidence. This will allow us to compare LLM evaluations with human expert assessments. +- **UI Development**: Develop a user-friendly interface for interacting with the CLLM tool. This could include features for uploading papers, viewing extracted claims, evaluating claims, and visualizing the comparison between LLM and peer review evaluations. +- **Incorporate into StructSense Agentic Framework**: CLLM provides a concrete, well-scoped pipeline for assertion extraction and evaluation that allows us to move quickly and populate the knowledge graph. As this stabilizes, we can selectively extract the components and prompting strategies that prove effective and reimplement them as configurable agents within StructSense. This approach allows us to benefit from CLLM’s task-specific design today while preserving StructSense as the long-term, general-purpose agent workflow system. + +## CLLM CLI Integration Architecture + +```mermaid +flowchart TB + %% Inputs + A1[Input: PDF] --> P[PDF Parsing Layer] + A2[Input: XML/Text] + + %% CLLM core + C1[CLLM: Extract Claims] + C2[CLLM: LLM Evaluation of Claims] + C3["CLLM: Peer Review Evaluation (Optional)"] + C4[CLLM: Compare LLM and Peer Review Evaluations] + + %% Process CLLM outputs + P1["Filter Output: (1) Review Status (2) If applicable, agreement between LLM and peer review evaluations"] + P2["Format Output: Map output to BICAN Assertion/Evidence Model"] + P3["Ingest into Knowledge Graph"] + P --> C1 + A2 --> C1 + C1 --> C2 + C1 --> C3 + C2 --> C4 + C3 --> C4 + C4 --> P1 + P1 --> P2 + P2 --> P3 + +``` + +## CLLM Output + +| Paper | Extraction Metrics | Evaluation Metrics | Output Files | Notes | +|-------|---------|---------|---------|-------| +| [bioRxiv 2024.04.22.590194v2](https://www.biorxiv.org/content/10.1101/2024.04.22.590194v2) | **Number of Extracted Claims:** 142
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.523
**Processing Time:** 305.99 seconds | **Number of Result Groups:** 25
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.408
**Processing Time:** 96.29 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1Piivsw6jn1rZxewQ5RjInG7lc_RX0qQz) |
+| [bioRxiv 2025.12.02.691876](https://www.biorxiv.org/content/10.64898/2025.12.02.691876v1) | **Number of Extracted Claims:** 79
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.485
**Processing Time:** 173.44 seconds | **Number of Result Groups:** 21
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.454
**Processing Time:** 94.98 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1_Aed64qQhg-_ZQNTNbW9UZGwEBbfSMXn) | |
+| [bioRxiv 2025.12.31.697063](https://www.biorxiv.org/content/10.64898/2025.12.31.697063v1) | **Number of Extracted Claims:** 105
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.612
**Processing Time:** 276.13 seconds | **Number of Result Groups:** 18
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.525
**Processing Time:** 99.42 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1InpnQDWyZKD-dg5Q7b-XyXGHtofP-h9-) | Warning when running `cllm eval`: `UserWarning: [STAGE 2: LLM Group Claims Into Results] Prompt may exceed 100000 tokens! Prompt length estimate: 111524 tokens.` |
+| [bioRxiv 2025.12.19.695583](https://www.biorxiv.org/content/10.64898/2025.12.19.695583v1) | | | | Error when running `cllm extract` (tried running twice): `Error extracting claims: Failed to parse LLM response as JSON: Expecting ',' delimiter: line 870 column 217 (char 64713)` | \ No newline at end of file From 38ea0b66347562493c8ae8e696d77c195efe0ab3 Mon Sep 17 00:00:00 2001 From: puja-trivedi <44144244+puja-trivedi@users.noreply.github.com> Date: Fri, 6 Feb 2026 17:07:42 -0800 Subject: [PATCH 2/4] Update doc/design/cllm_integration.md Removed trailing space in the 'MVP' heading. Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- doc/design/cllm_integration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/cllm_integration.md b/doc/design/cllm_integration.md index 558b0c3..1a6c613 100644 --- a/doc/design/cllm_integration.md +++ b/doc/design/cllm_integration.md @@ -9,7 +9,7 @@ CLLM implements a 4-stage workflow for scientific claim verification: 3. **Peer Review Evaluation**: Group claims based on peer review commentary 4. **Compare Results**: Compare LLM and peer review evaluations -## MVP +## MVP Ingest assertions and evidence from scientific papers into the knowledge graph using CLLM, with the following steps: 1. **Input**: Accept PDF and XML/Text inputs of scientific papers. 2. **CLLM Processing**: Run the CLLM workflow to extract claims and evaluate them using LLM review. From 39cf50b5d3d233b23ede6915889c3946da738451 Mon Sep 17 00:00:00 2001 From: puja-trivedi <44144244+puja-trivedi@users.noreply.github.com> Date: Fri, 6 Feb 2026 17:08:14 -0800 Subject: [PATCH 3/4] Update doc/design/cllm_integration.md corrected the table formatting for consistency and proper rendering. Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- doc/design/cllm_integration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/design/cllm_integration.md b/doc/design/cllm_integration.md index 1a6c613..3f7221d 100644 --- a/doc/design/cllm_integration.md +++ b/doc/design/cllm_integration.md @@ -58,7 +58,7 @@ flowchart TB | Paper | Extraction Metrics | Evaluation Metrics | Output Files | Notes | |-------|---------|---------|---------|-------| -| [bioRxiv 2024.04.22.590194v2](https://www.biorxiv.org/content/10.1101/2024.04.22.590194v2) | **Number of Extracted Claims:** 142
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.523
**Processing Time:** 305.99 seconds | **Number of Result Groups:** 25
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.408
**Processing Time:** 96.29 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1Piivsw6jn1rZxewQ5RjInG7lc_RX0qQz) |
-| [bioRxiv 2025.12.02.691876](https://www.biorxiv.org/content/10.64898/2025.12.02.691876v1) | **Number of Extracted Claims:** 79
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.485
**Processing Time:** 173.44 seconds | **Number of Result Groups:** 21
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.454
**Processing Time:** 94.98 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1_Aed64qQhg-_ZQNTNbW9UZGwEBbfSMXn) | |
-| [bioRxiv 2025.12.31.697063](https://www.biorxiv.org/content/10.64898/2025.12.31.697063v1) | **Number of Extracted Claims:** 105
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.612
**Processing Time:** 276.13 seconds | **Number of Result Groups:** 18
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.525
**Processing Time:** 99.42 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1InpnQDWyZKD-dg5Q7b-XyXGHtofP-h9-) | Warning when running `cllm eval`: `UserWarning: [STAGE 2: LLM Group Claims Into Results] Prompt may exceed 100000 tokens! Prompt length estimate: 111524 tokens.` |
+| [bioRxiv 2024.04.22.590194v2](https://www.biorxiv.org/content/10.1101/2024.04.22.590194v2) | **Number of Extracted Claims:** 142
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.523
**Processing Time:** 305.99 seconds | **Number of Result Groups:** 25
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.408
**Processing Time:** 96.29 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1Piivsw6jn1rZxewQ5RjInG7lc_RX0qQz) | | +| [bioRxiv 2025.12.02.691876](https://www.biorxiv.org/content/10.64898/2025.12.02.691876v1) | **Number of Extracted Claims:** 79
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.485
**Processing Time:** 173.44 seconds | **Number of Result Groups:** 21
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.454
**Processing Time:** 94.98 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1_Aed64qQhg-_ZQNTNbW9UZGwEBbfSMXn) | | +| [bioRxiv 2025.12.31.697063](https://www.biorxiv.org/content/10.64898/2025.12.31.697063v1) | **Number of Extracted Claims:** 105
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.612
**Processing Time:** 276.13 seconds | **Number of Result Groups:** 18
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.525
**Processing Time:** 99.42 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1InpnQDWyZKD-dg5Q7b-XyXGHtofP-h9-) | Warning when running `cllm eval`: `UserWarning: [STAGE 2: LLM Group Claims Into Results] Prompt may exceed 100000 tokens! Prompt length estimate: 111524 tokens.` | | [bioRxiv 2025.12.19.695583](https://www.biorxiv.org/content/10.64898/2025.12.19.695583v1) | | | | Error when running `cllm extract` (tried running twice): `Error extracting claims: Failed to parse LLM response as JSON: Expecting ',' delimiter: line 870 column 217 (char 64713)` | \ No newline at end of file From 8e47cdddf54cd3abc9e544acd18693fd9c2bbf64 Mon Sep 17 00:00:00 2001 From: puja-trivedi <44144244+puja-trivedi@users.noreply.github.com> Date: Mon, 9 Feb 2026 11:04:49 -0800 Subject: [PATCH 4/4] Update cllm_integration.md updated google drive links for output --- doc/design/cllm_integration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/design/cllm_integration.md b/doc/design/cllm_integration.md index 3f7221d..53dfc57 100644 --- a/doc/design/cllm_integration.md +++ b/doc/design/cllm_integration.md @@ -58,7 +58,7 @@ flowchart TB | Paper | Extraction Metrics | Evaluation Metrics | Output Files | Notes | |-------|---------|---------|---------|-------| -| [bioRxiv 2024.04.22.590194v2](https://www.biorxiv.org/content/10.1101/2024.04.22.590194v2) | **Number of Extracted Claims:** 142
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.523
**Processing Time:** 305.99 seconds | **Number of Result Groups:** 25
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.408
**Processing Time:** 96.29 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1Piivsw6jn1rZxewQ5RjInG7lc_RX0qQz) | | -| [bioRxiv 2025.12.02.691876](https://www.biorxiv.org/content/10.64898/2025.12.02.691876v1) | **Number of Extracted Claims:** 79
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.485
**Processing Time:** 173.44 seconds | **Number of Result Groups:** 21
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.454
**Processing Time:** 94.98 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1_Aed64qQhg-_ZQNTNbW9UZGwEBbfSMXn) | | -| [bioRxiv 2025.12.31.697063](https://www.biorxiv.org/content/10.64898/2025.12.31.697063v1) | **Number of Extracted Claims:** 105
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.612
**Processing Time:** 276.13 seconds | **Number of Result Groups:** 18
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.525
**Processing Time:** 99.42 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1InpnQDWyZKD-dg5Q7b-XyXGHtofP-h9-) | Warning when running `cllm eval`: `UserWarning: [STAGE 2: LLM Group Claims Into Results] Prompt may exceed 100000 tokens! Prompt length estimate: 111524 tokens.` | -| [bioRxiv 2025.12.19.695583](https://www.biorxiv.org/content/10.64898/2025.12.19.695583v1) | | | | Error when running `cllm extract` (tried running twice): `Error extracting claims: Failed to parse LLM response as JSON: Expecting ',' delimiter: line 870 column 217 (char 64713)` | \ No newline at end of file +| [bioRxiv 2024.04.22.590194v2](https://www.biorxiv.org/content/10.1101/2024.04.22.590194v2) | **Number of Extracted Claims:** 142
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.523
**Processing Time:** 305.99 seconds | **Number of Result Groups:** 25
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.408
**Processing Time:** 96.29 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1HR-HIhbM4cw1nT-4iByaHtOqDdDcN18K) | | +| [bioRxiv 2025.12.02.691876](https://www.biorxiv.org/content/10.64898/2025.12.02.691876v1) | **Number of Extracted Claims:** 79
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.485
**Processing Time:** 173.44 seconds | **Number of Result Groups:** 21
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.454
**Processing Time:** 94.98 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1kdqbDrq_6Y4Es8Yf6gXco2fA8QDELVNp) | | +| [bioRxiv 2025.12.31.697063](https://www.biorxiv.org/content/10.64898/2025.12.31.697063v1) | **Number of Extracted Claims:** 105
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.612
**Processing Time:** 276.13 seconds | **Number of Result Groups:** 18
**Model:** claude-sonnet-4-5-20250929
**Total Cost:** $0.525
**Processing Time:** 99.42 seconds | [Output Files](https://drive.google.com/drive/u/0/folders/1U_C1CIxYrpSaco4P5HGyVdoiEJN3_1AZ) | Warning when running `cllm eval`: `UserWarning: [STAGE 2: LLM Group Claims Into Results] Prompt may exceed 100000 tokens! Prompt length estimate: 111524 tokens.` | +| [bioRxiv 2025.12.19.695583](https://www.biorxiv.org/content/10.64898/2025.12.19.695583v1) | | | | Error when running `cllm extract` (tried running twice): `Error extracting claims: Failed to parse LLM response as JSON: Expecting ',' delimiter: line 870 column 217 (char 64713)` |