Pipeline Hangs During Generation

[joesan]

I tried this earlier today, but any idea why the pipeline hangs?

OPENAI_API_KEY=sk-xxx-xxx-xxx-xxx uv run python -m attractor_pipeline.cli run scala_dev.dot --no-tools --provider openai --model gpt-4.1-mini
Pipeline: POIMicroservicePipeline
Goal: Implement Scala POI Microservice
Nodes: 16, Edges: 20

Provider: openai (gpt-4.1-mini)
Environment: Local
Backend: DirectLLM (no tools)

Executing pipeline...
----------------------------------------

Here is the dot file that I tried:

digraph POIMicroservicePipeline {
    graph [goal="Implement Scala POI Microservice", project_root="./services/poi-service"]

    // Start and exit
    start  [shape=Mdiamond, label="Start"]
    exit   [shape=Msquare, label="Exit"]

    // Stage 1: Scaffold
    scaffold_service [label="Scaffold Scala Microservice", prompt="Generate a Scala microservice project using Akka HTTP (or Play). Include build.sbt, main class, project structure, and a placeholder POIController."]
    review_scaffold  [shape=hexagon, label="Human Check: Scaffold"]

    // Stage 2: Endpoints
    implement_create [label="POST /poi", prompt="Generate Scala code for POST /poi endpoint: accept JSON id, name, latitude, longitude. Return created POI."]
    implement_read   [label="GET /poi/{id}", prompt="Generate Scala code for GET /poi/{id} endpoint: return POI or 404."]
    implement_update [label="PUT /poi/{id}", prompt="Generate Scala code for PUT /poi/{id} endpoint: update POI or 404."]
    implement_delete [label="DELETE /poi/{id}", prompt="Generate Scala code for DELETE /poi/{id} endpoint: remove POI, return 204 or 404."]
    review_endpoints [shape=hexagon, label="Human Check: Endpoints"]

    // Stage 2b: OpenAPI
    openapi_spec    [label="Generate OpenAPI/Swagger Spec", prompt="Generate a Swagger/OpenAPI specification (YAML or JSON) for all POI endpoints. Include request/response models, field types, and examples."]
    review_openapi  [shape=hexagon, label="Human Check: OpenAPI Spec"]

    // Stage 3: Persistence
    persistence       [label="Add Persistence Layer", prompt="Implement PostgreSQL integration using Slick. Include POI table schema and DAO for CRUD operations. Ensure endpoints persist and retrieve data correctly."]
    review_persistence [shape=hexagon, label="Human Check: Persistence"]

    // Stage 4: Testing
    test_unit        [label="Unit Tests", prompt="Generate Scala unit tests for each endpoint and DAO method using ScalaTest. Include validation of edge cases and errors."]
    test_integration [label="Integration Tests", prompt="Generate Scala integration tests: run the microservice, exercise endpoints, and assert correct database behavior."]
    review_testing   [shape=hexagon, label="Human Check: Testing"]

    // Stage 5: Commit & Push
    // merge [label="Commit & Push Initial Project", prompt="Commit all scaffold, endpoint, OpenAPI spec, persistence, and test code to the main branch and push to GitHub."]

    // Workflow edges
    start -> scaffold_service -> review_scaffold
    review_scaffold -> implement_create -> implement_read -> implement_update -> implement_delete -> review_endpoints
    review_endpoints -> openapi_spec -> review_openapi -> persistence -> review_persistence
    review_persistence -> test_unit -> test_integration -> review_testing -> exit
    //review_persistence -> test_unit -> test_integration -> review_testing -> merge -> exit

    // Retry paths if human requests changes
    review_scaffold -> scaffold_service [label="Revise", condition="Human requests changes"]
    review_endpoints -> implement_create [label="Revise", condition="Human requests changes"]
    review_openapi -> openapi_spec [label="Revise", condition="Human requests changes"]
    review_persistence -> persistence [label="Revise", condition="Human requests changes"]
    review_testing -> test_unit [label="Revise", condition="Human requests changes"]
}

Any clue as to what is happening?

[samueljklee]

Hi @joesan, thanks for reporting this and for including the DOT file -- made it straightforward to track down.

What's happening

shape=hexagon maps to the Manager handler (sub-pipeline orchestrator), which expects a child_graph attribute. Your review nodes need shape=house, which maps to the Human handler (approval gate).

Without the correct shape, the handler fails silently on every attempt, retries 50 times per node, then advances -- all without printing anything. That's why it looks like a hang.

The fix

Two changes to your DOT file:

1. shape=hexagon -> shape=house on all review nodes
2. Label the forward edges (e.g., label="Approve") from review nodes

The second point matters because the Human handler uses outgoing edge labels as options for the interviewer. In non-interactive CLI mode, auto-approve picks the first option. If the only labeled edge is "Revise", it picks that and loops back. Adding "Approve" to the forward edge gives it a correct option to pick.

Here's an updated version of your pipeline:

digraph POIMicroservicePipeline {
    graph [goal="Implement Scala POI Microservice", project_root="./services/poi-service"]

    start  [shape=Mdiamond, label="Start"]
    exit   [shape=Msquare, label="Exit"]

    scaffold_service [label="Scaffold Scala Microservice", prompt="Generate a Scala microservice project using Akka HTTP (or Play). Include build.sbt, main class, project structure, and a placeholder POIController."]
    review_scaffold  [shape=house, label="Human Check: Scaffold"]

    implement_create [label="POST /poi", prompt="Generate Scala code for POST /poi endpoint: accept JSON id, name, latitude, longitude. Return created POI."]
    implement_read   [label="GET /poi/{id}", prompt="Generate Scala code for GET /poi/{id} endpoint: return POI or 404."]
    implement_update [label="PUT /poi/{id}", prompt="Generate Scala code for PUT /poi/{id} endpoint: update POI or 404."]
    implement_delete [label="DELETE /poi/{id}", prompt="Generate Scala code for DELETE /poi/{id} endpoint: remove POI, return 204 or 404."]
    review_endpoints [shape=house, label="Human Check: Endpoints"]

    openapi_spec    [label="Generate OpenAPI/Swagger Spec", prompt="Generate a Swagger/OpenAPI specification (YAML or JSON) for all POI endpoints. Include request/response models, field types, and examples."]
    review_openapi  [shape=house, label="Human Check: OpenAPI Spec"]

    persistence       [label="Add Persistence Layer", prompt="Implement PostgreSQL integration using Slick. Include POI table schema and DAO for CRUD operations. Ensure endpoints persist and retrieve data correctly."]
    review_persistence [shape=house, label="Human Check: Persistence"]

    test_unit        [label="Unit Tests", prompt="Generate Scala unit tests for each endpoint and DAO method using ScalaTest. Include validation of edge cases and errors."]
    test_integration [label="Integration Tests", prompt="Generate Scala integration tests: run the microservice, exercise endpoints, and assert correct database behavior."]
    review_testing   [shape=house, label="Human Check: Testing"]

    // Workflow edges (forward edges labeled for auto-approve)
    start -> scaffold_service
    scaffold_service -> review_scaffold
    review_scaffold -> implement_create [label="Approve"]
    implement_create -> implement_read -> implement_update -> implement_delete
    implement_delete -> review_endpoints
    review_endpoints -> openapi_spec [label="Approve"]
    openapi_spec -> review_openapi
    review_openapi -> persistence [label="Approve"]
    persistence -> review_persistence
    review_persistence -> test_unit [label="Approve"]
    test_unit -> test_integration
    test_integration -> review_testing
    review_testing -> exit [label="Approve"]

    // Retry paths
    review_scaffold -> scaffold_service [label="Revise"]
    review_endpoints -> implement_create [label="Revise"]
    review_openapi -> openapi_spec [label="Revise"]
    review_persistence -> persistence [label="Revise"]
    review_testing -> test_unit [label="Revise"]
}

We tested this locally -- validates clean and runs to completion with all 16 nodes executing exactly once.

On our end

We'll be adding a validation rule to catch this at attractor validate time, and execution progress output so pipelines don't run in silence.

[joesan]

Thanks for getting back on this. I somehow keep experiencing the same issue even with the modified dot file yor provided:

Pipeline: POIMicroservicePipeline
Goal: Implement Scala POI Microservice
Nodes: 16, Edges: 20

Provider: openai (gpt-4.1-mini)
Environment: Local
Backend: DirectLLM (no tools)

Executing pipeline...
----------------------------------------

[samueljklee]

Hey @joesan -- can you pull the latest? We just shipped a --verbose flag that shows real-time progress:

attractor run scala_dev.dot --no-tools --provider openai --model gpt-4.1-mini --verbose

We ran your exact pipeline with the corrected DOT file and it completed in ~2.5 minutes -- some nodes take 20-40s for the API call. Without --verbose, that's 2.5 minutes of silence, which is why it looks like a hang.

With --verbose you'll see each node as it executes:

[event] Stage 'scaffold_service' [1] started
[event] Stage 'scaffold_service' [1] completed in 17.1s
[event] Stage 'implement_create' [3] started
[event] Stage 'implement_create' [3] completed in 9.3s
...

Let us know if you're still stuck after updating.

[joesan]

With verbose, I now see the progress, but I'm still stuck at the scaffold_service stage and it keeps retrying. It would be informative if it also showed me why it is retrying:

Provider: openai (gpt-4.1-mini)
Environment: Local
Backend: DirectLLM (no tools)

Executing pipeline...
----------------------------------------
  [event] Pipeline 'POIMicroservicePipeline' started (id=c09eae7e)
  [event] Stage 'start' [0] started
  [event] Stage 'start' [0] completed in 0.0s
  [event] Stage 'scaffold_service' [1] started
  [event] Stage 'scaffold_service' [1] retrying (attempt 1, delay 0.0s)
  [event] Stage 'scaffold_service' [2] started
  [event] Stage 'scaffold_service' [2] retrying (attempt 2, delay 0.0s)
  [event] Stage 'scaffold_service' [3] started
  [event] Stage 'scaffold_service' [3] retrying (attempt 3, delay 0.0s)
  [event] Stage 'scaffold_service' [4] started

May be the error is hidden somewhere in the logs?

[samueljklee]

Great feedback @joesan -- we've updated attractor to show the error reason while it's retrying. Pull latest and give it a try:

[event] Stage 'scaffold_service' [1] retrying (attempt 1, delay 0.0s): Invalid API key: sk-xxx...xxx

That should tell you immediately what's going wrong with the API call.

We also proposed this as a spec enhancement upstream: strongdm/attractor#4.

[joesan]

Yes, I have the error message that now shows up. Thanks for the fix!