Add LLM workflow planner / builder and field mapping #244

muthukumaranR · 2025-10-02T11:20:49Z

Overview

This PR adds a LLM-based workflow planner that converts natural language research requests into executable workflows (compatible with AKD Backend) with automatic data routing between agents.

Problem

When workflows need multiple agent interactions (which are composed by LLM) - and when chaining agents in a workflow (e.g., deep_search → gap_analysis), we need to:

Convert LLM plans into executable workflow JSON
Route data from one agent's outputs to the next agent's inputs
Handle field name mismatches (e.g., results → search_results)
Support both manual mappings and LLM-generated mappings

Solution

LLM Workflow Planner

Interactive conversational planner that:

Understands research requirements through dialogue
Selects appropriate agents from registry
Extracts agent inputs using full conversation context:
- Initial user request
- Conversation history and refinements
- Research goal and workflow plan
- Agent selection reasoning
- Field schemas and dependencies

7-Phase Conversation Flow:

Initial Requirements - Understand research goal
Goal Clarification - Refine objectives
Agent Selection - Choose appropriate agents
IO Specification - Define inputs/outputs
Workflow Construction - Build execution plan
Validation - Verify completeness
Finalization - Generate executable workflow

Workflow Builder

Transforms high-level plans into executable workflows:

Reads agent schemas from the registry (PR feat: Agent Registry - Centralized Agent Discovery & Schema Management #236)
Generates JSONPath expressions for runtime data access
Creates complete workflow JSON ready for execution

Field Mapping System

Three-tier strategy for mapping fields:

Explicit: Human-defined in field_mappings.json
Exact Match: Auto-match when names are identical
LLM Semantic: AI-generated with confidence scoring

Why JSONPath?

Problem: At runtime, how does gap_analysis get deep_search's results?

Without JSONPath (hardcoded):

# Custom logic for every agent pair
gap_analysis_input = {
    "search_results": deep_search_output["results"]
}

With JSONPath (declarative):

{
  "type": "gap_analysis",
  "io_map": {
    "search_results": "$.deep_search.outputs.results"
  }
}

The executor reads the JSONPath and fetches data from the runtime state - no custom code needed per agent pair.

Implementation

Core Components

LLM Planner (NEW):

llm_planner.py: Conversational workflow planning
LLMWorkflowPlanner: Main planner class using LiteLLMInstructorBaseAgent (multi-provider support)
InteractivePlannerSession: Manages conversation state
Input extraction using full conversation context
Automatic message trimming to prevent token limit errors

Workflow Builder:

format_builder.py: Data models (WorkflowFormat, WorkflowNode, etc.)
workflow_builder.py: Workflow generation with auto-generated io_map
structures.py: Shared data structures (WorkflowPlan, AgentSuggestion)

Field Mapping:

field_mapping_registry.py: Mapping storage and resolution
field_mapping_generator.py: LLM-based semantic mapping

Documentation (NEW):

akd/planner/README.md: Comprehensive documentation
Architecture overview with diagrams
End-to-end workflow examples
Configuration guide

Demo (NEW):

scripts/demo_planner.py: Interactive CLI demo
Modes: interactive, quick, agents, sample

Validation System

WorkflowFormat Validation (format_builder.py):

validate_io_map(): Validates JSONPath references (format, node existence)
validate_edges(): Validates graph structure (START/END, orphaned nodes)
validate(): Validation with categorized results
get_node_summary(): Workflow statistics (nodes, edges, io_map entries)

Conversation Control (llm_planner.py):

Context-aware ready_to_generate validator prevents infinite loops
Auto-corrects when contradiction exists (plan ready, no question, flag false)
Preserves LLM decision-making for confirmation requests and clarifications

Enum Normalization:

Field validators normalize enum values to lowercase
Handles LLM uppercase outputs

io_map Generation Example

# Input: field mapping metadata
"deep_search->gap_analysis": {
  "search_results": "results"
}

# Output: JSONPath for runtime
io_map = {
  "search_results": "$.deep_search.outputs.results"
}

Tests

79 tests

# Run all planner tests
pytest tests/planner/ -v -o addopts=""

# Breakdown:
# - test_workflow_builder.py:    15 tests
# - test_field_mapping.py:        14 tests
# - test_registry.py:             23 tests
# - test_llm_planner.py:          16 tests
# - test_workflow_format.py:      11 tests

Test Coverage:

Workflow building and serialization
Field mapping (explicit, LLM, exact match)
Agent registry integration
LLM planner conversation flow
Input extraction with full context
Workflow format validation (io_map, edges, structure)
Error handling and edge cases

Files Changed

Added:

akd/planner/format_builder.py
akd/planner/workflow_builder.py
akd/planner/field_mapping_registry.py
akd/planner/field_mapping_generator.py
akd/planner/structures.py
akd/planner/README.md
akd/mapping/field_mappings.json
scripts/minimal_planner.py
tests/planner/test_workflow_builder.py
tests/planner/test_field_mapping.py
tests/planner/test_workflow_format.py

Modified:

akd/planner/init.py (added exports)
akd/agents/search/_base.py (added report field)
akd/agents/search/deep_search.py (populate report)
akd/mapping/agent_registry.json (updated schema)

Usage

Quick Start - LLM Planner

from akd.planner.llm_planner import quick_plan

# Quick planning (non-interactive)
session = await quick_plan("Find recent papers on AlphaFold protein structure prediction")
response = await session.start()
workflow = await session.generate_workflow()

# Save to file
workflow.save_to_file("my_workflow.json")

Interactive Planning (NEW)

from akd.planner.llm_planner import create_planner

# Create planner
planner = await create_planner()

# Start interactive session
session = await planner.plan_workflow("Find papers on AlphaFold and identify research gaps")
response = await session.start()

# Conversation loop
while not response.ready_to_generate:
    print(f"Planner: {response.message}")
    user_input = input("Your response: ")
    response = await session.respond(user_input)

# Generate workflow
workflow = await session.generate_workflow()
workflow.save_to_file("alphafold_workflow.json")

Backend Integration Example

scripts/minimal_planner.py demonstrates backend/frontend integration:

python3 scripts/minimal_planner.py "Find papers on AlphaFold and identify research gaps"

Direct WorkflowBuilder Usage

from akd.planner import (
    get_agent_registry,
    WorkflowBuilder,
    WorkflowPlan,
    AgentSuggestion,
)

# Initialize (uses registry from PR #236)
registry = get_agent_registry()
builder = WorkflowBuilder(registry)

# Create plan
plan = WorkflowPlan(
    workflow_description="Find papers then analyze gaps",
    research_goal="Identify research gaps in AlphaFold",
    suggested_agents=[
        AgentSuggestion(
            agent_id="deep_search",
            agent_name="Deep Literature Search",
            reason="Search for papers",
            confidence=0.95,
        ),
        AgentSuggestion(
            agent_id="gap_analysis",
            agent_name="Gap Analysis",
            reason="Analyze gaps",
            confidence=0.90,
        )
    ],
    estimated_complexity="medium"
)

# Build workflow with auto-generated io_map
workflow = builder.build(plan, {
    "deep_search": {"query": "AlphaFold"}
})

# Result: gap_analysis.search_results auto-mapped to $.deep_search.outputs.results
print(workflow.to_json())

Interactive Demo

Try the interactive CLI:

# Interactive planning session
python scripts/demo_planner.py interactive

# Quick planning
python scripts/demo_planner.py quick "Find papers on protein folding"

# Show available agents
python scripts/demo_planner.py agents

# Sample workflow format
python scripts/demo_planner.py sample

Documentation

See comprehensive documentation at akd/planner/README.md.

Enabled Workflows

deep_search → gap_analysis
code_search → gap_analysis
deep_search → code_search → gap_analysis

github-actions · 2025-10-02T11:31:41Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 380
Failed: 4
Skipped: 6
Warnings: 180
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: 88a746d

📋 Full coverage report and logs are available in the workflow run.

…nd update related agent logic - Introduced a new field `report` in `LitSearchAgentOutputSchema` to store synthesized research reports. - Updated `DeepLitSearchAgent` to pass the `research_report` to the new `report` field. - Modified `agent_registry.json` to include the new `report` field in the agent configuration. - Refactored test cases in `deep_search_test.py` to reflect changes in the output schema and ensure proper validation of the report field.

- define classes for building workflow definitions in the AKD framework. - Implement `WorkflowField`, `WorkflowNodeIO`, `WorkflowNode`, `WorkflowEdge`, and `WorkflowFormat` classes for structured workflow representation. - Added methods for custom serialization, JSON export, and file saving for workflow definitions. - Included functions to create and load workflows from dictionaries and JSON files, respectively.

- Introduced `WorkflowBuilder` class to construct `WorkflowFormat` from `WorkflowPlan` with runtime data flow. - Implemented a three-tier field mapping strategy: explicit mappings, exact name matches, and LLM-generated mappings. - Added methods for building nodes and edges, validating agent existence, and identifying unmapped fields for LLM generation. - Integrated logging for better traceability of mapping processes and warnings for missing agents or fields.

github-actions · 2025-10-02T17:53:00Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 381
Failed: 3
Skipped: 6
Warnings: 180
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: 8015777

📋 Full coverage report and logs are available in the workflow run.

github-actions · 2025-10-02T20:00:04Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 381
Failed: 3
Skipped: 6
Warnings: 179
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: 82782f0

📋 Full coverage report and logs are available in the workflow run.

NISH1001

Some comments: Some are nitpick. Some are improvements.

akd/agents/search/_base.py

akd/planner/format_builder.py

akd/planner/workflow_builder.py

NISH1001 · 2025-10-03T01:43:47Z

REsolving some of these comments myself.

- ALos use pydantic-v2 style model config `model_config` intead of `class Config`

- Bugfix `report` field access

github-actions · 2025-10-03T02:25:15Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 380
Failed: 4
Skipped: 6
Warnings: 178
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: ee1da70

📋 Full coverage report and logs are available in the workflow run.

github-actions · 2025-10-03T02:32:19Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 382
Failed: 2
Skipped: 6
Warnings: 179
Coverage: 79%

Branch: feature/planner-builder
PR: #244
Commit: 80bc1d4

📋 Full coverage report and logs are available in the workflow run.

akd/planner/workflow_builder.py

github-actions · 2025-10-03T02:55:47Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 382
Failed: 2
Skipped: 6
Warnings: 179
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: 8123b83

📋 Full coverage report and logs are available in the workflow run.

- FieldMappingRegistry: Three-tier mapping resolution (explicit, exact match, LLM) - FieldMappingGenerator: LLM-based semantic field mapper - Required by WorkflowBuilder for intelligent field mapping between agents

github-actions · 2025-10-03T13:25:54Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 382
Failed: 2
Skipped: 6
Warnings: 179
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: 48e43d9

📋 Full coverage report and logs are available in the workflow run.

… auto-set ready_to_generate based on workflow_plan existence, and rename plan_workflow to init_planner_session for clarity in LLMWorkflowPlanner class. Enhance workflow validation logging in InteractivePlannerSession.

…r.py for improved readability. Update method calls to use init_planner_session for clarity and ensure consistent string formatting throughout the file.

…_format.py to ensure proper detection of invalid node references, JSONPath formats, orphaned nodes, and edge cases. Enhance coverage for both strict and non-strict validation scenarios.

github-actions · 2025-10-06T22:42:29Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 438
Failed: 2
Skipped: 6
Warnings: 179
Coverage: 79%

Branch: feature/planner-builder
PR: #244
Commit: 920da35

📋 Full coverage report and logs are available in the workflow run.

github-actions · 2025-10-06T22:43:22Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 427
Failed: 2
Skipped: 6
Warnings: 180
Coverage: 79%

Branch: feature/planner-builder
PR: #244
Commit: b6ba676

📋 Full coverage report and logs are available in the workflow run.

- `conversation_history: list[dict[str, str]]` is added to INteractivePlannerSession - So, that externally we can inject the history for resuming

github-actions · 2025-10-07T16:15:37Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 438
Failed: 2
Skipped: 6
Warnings: 179
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: be641fd

📋 Full coverage report and logs are available in the workflow run.

akd/planner/field_mapping_registry.py

akd.utils.get_akd_root

github-actions · 2025-10-07T20:44:41Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 448
Failed: 2
Skipped: 6
Warnings: 179
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: 57aa68c

📋 Full coverage report and logs are available in the workflow run.

github-actions · 2025-10-07T20:48:56Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 449
Failed: 1
Skipped: 6
Warnings: 180
Coverage: 79%

Branch: feature/planner-builder
PR: #244
Commit: d1d718e

📋 Full coverage report and logs are available in the workflow run.

github-actions · 2025-10-10T15:11:56Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 448
Failed: 2
Skipped: 6
Warnings: 179
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: 0fcd94f

📋 Full coverage report and logs are available in the workflow run.

- `LLMMappingEntry.generated_at` now uses default factory

github-actions · 2025-10-10T15:31:49Z

❌ Tests failed (exit code: 1)

📊 Test Results

Passed: 449
Failed: 1
Skipped: 6
Warnings: 184
Coverage: 80%

Branch: feature/planner-builder
PR: #244
Commit: aadd26d

📋 Full coverage report and logs are available in the workflow run.

muthukumaranR temporarily deployed to integration October 2, 2025 11:20 — with GitHub Actions Inactive

muthukumaranR mentioned this pull request Oct 2, 2025

Workflow Builder + Field Mapping System #241

Closed

muthukumaranR added 4 commits October 2, 2025 12:41

Add field mappings configuration for agent interactions

c1a9a00

muthukumaranR force-pushed the feature/planner-builder branch from a278d60 to 91e5b08 Compare October 2, 2025 17:41

muthukumaranR temporarily deployed to integration October 2, 2025 17:41 — with GitHub Actions Inactive

muthukumaranR changed the title ~~feat(planner): Add workflow builder with field mapping system~~ Add workflow builder with field mapping system Oct 2, 2025

NISH1001 self-requested a review October 2, 2025 19:46

Refactor deep search tests to reflect changed output schema for report

b3bed4f

muthukumaranR temporarily deployed to integration October 2, 2025 19:48 — with GitHub Actions Inactive

NISH1001 requested changes Oct 2, 2025

View reviewed changes

Refactor type hint for workflownode to use ne wpython style

3b93876

- ALos use pydantic-v2 style model config `model_config` intead of `class Config`

NISH1001 temporarily deployed to integration October 3, 2025 02:15 — with GitHub Actions Inactive

Bugfix tests for deep search agent

1919af1

- Bugfix `report` field access

NISH1001 temporarily deployed to integration October 3, 2025 02:21 — with GitHub Actions Inactive

Refactor WorkflowFormat to have class methods

6f9e874

NISH1001 temporarily deployed to integration October 3, 2025 02:44 — with GitHub Actions Inactive

NISH1001 requested changes Oct 3, 2025

View reviewed changes

akd/planner/workflow_builder.py Show resolved Hide resolved

akd/planner/workflow_builder.py Outdated Show resolved Hide resolved

Add field mapping registry and generator modules

c2df25f

- FieldMappingRegistry: Three-tier mapping resolution (explicit, exact match, LLM) - FieldMappingGenerator: LLM-based semantic field mapper - Required by WorkflowBuilder for intelligent field mapping between agents

muthukumaranR temporarily deployed to integration October 3, 2025 13:14 — with GitHub Actions Inactive

muthukumaranR added 3 commits October 6, 2025 17:27

Refactor import statements and enhance code formatting in demo_planne…

a1ddc17

…r.py for improved readability. Update method calls to use init_planner_session for clarity and ensure consistent string formatting throughout the file.

Add minimal interactive planner script for backend/frontend integration.

afdd23e

muthukumaranR temporarily deployed to integration October 6, 2025 22:32 — with GitHub Actions Inactive

Add unit tests for WorkflowFormat validation methods in test_workflow…

2bc194a

…_format.py to ensure proper detection of invalid node references, JSONPath formats, orphaned nodes, and edge cases. Enhance coverage for both strict and non-strict validation scenarios.

muthukumaranR temporarily deployed to integration October 6, 2025 22:32 — with GitHub Actions Inactive

Add conversation_history parameter to planner interactive session

01c4c5b

- `conversation_history: list[dict[str, str]]` is added to INteractivePlannerSession - So, that externally we can inject the history for resuming

NISH1001 temporarily deployed to integration October 7, 2025 16:02 — with GitHub Actions Inactive

NISH1001 requested changes Oct 7, 2025

View reviewed changes

akd/planner/field_mapping_registry.py Show resolved Hide resolved

NISH1001 requested changes Oct 7, 2025

View reviewed changes

akd/planner/field_mapping_registry.py Outdated Show resolved Hide resolved

akd/planner/field_mapping_registry.py Outdated Show resolved Hide resolved

Validate jsonpath in workflow builder for planner

d026aba

NISH1001 temporarily deployed to integration October 7, 2025 20:31 — with GitHub Actions Inactive

Change hardcoded path in field_mapping_registry to use

7f51a47

akd.utils.get_akd_root

NISH1001 temporarily deployed to integration October 7, 2025 20:36 — with GitHub Actions Inactive

Inject conversation_history to init_planner_session method

de2849c

NISH1001 temporarily deployed to integration October 10, 2025 14:59 — with GitHub Actions Inactive

Use default factory for llm-generated field mapping timestamp

3165065

- `LLMMappingEntry.generated_at` now uses default factory

NISH1001 temporarily deployed to integration October 10, 2025 15:20 — with GitHub Actions Inactive

NISH1001 approved these changes Oct 10, 2025

View reviewed changes

NISH1001 merged commit c759655 into develop Oct 10, 2025
1 check passed

NISH1001 deleted the feature/planner-builder branch October 10, 2025 19:01

muthukumaranR mentioned this pull request Oct 13, 2025

Separate Planner/Execution Flow in Accelerated Discovery Framework #9

Closed

muthukumaranR linked an issue Oct 14, 2025 that may be closed by this pull request

Planner roadmap #87

Closed

11 tasks

Add LLM workflow planner / builder and field mapping #244

Add LLM workflow planner / builder and field mapping #244

Uh oh!

Conversation

muthukumaranR commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Problem

Solution

LLM Workflow Planner

Workflow Builder

Field Mapping System

Why JSONPath?

Implementation

Core Components

Validation System

io_map Generation Example

Tests

Files Changed

Usage

Quick Start - LLM Planner

Interactive Planning (NEW)

Backend Integration Example

Direct WorkflowBuilder Usage

Interactive Demo

Documentation

Enabled Workflows

Uh oh!

github-actions bot commented Oct 2, 2025

📊 Test Results

Uh oh!

github-actions bot commented Oct 2, 2025

📊 Test Results

Uh oh!

github-actions bot commented Oct 2, 2025

📊 Test Results

Uh oh!

NISH1001 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NISH1001 commented Oct 3, 2025

Uh oh!

github-actions bot commented Oct 3, 2025

📊 Test Results

Uh oh!

github-actions bot commented Oct 3, 2025

📊 Test Results

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 3, 2025

📊 Test Results

Uh oh!

github-actions bot commented Oct 3, 2025

📊 Test Results

Uh oh!

github-actions bot commented Oct 6, 2025

📊 Test Results

Uh oh!

github-actions bot commented Oct 6, 2025

📊 Test Results

Uh oh!

github-actions bot commented Oct 7, 2025

📊 Test Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 7, 2025

📊 Test Results

muthukumaranR commented Oct 2, 2025 •

edited

Loading