feat: add Map Step Output Aggregation #217

jumski · 2025-09-17T07:08:39Z

This PR implements output aggregation for map steps, completing the critical functionality needed for map steps to pass their results to dependent steps.

What this adds:

Aggregates outputs from all map step tasks into a single array when the step completes
Preserves task_index ordering in the aggregated output array
Handles edge cases including empty maps (0 tasks) and NULL outputs from individual tasks
Optimizes map-to-map data flow to avoid unnecessary aggregation/decomposition cycles
Ensures dependent steps receive properly aggregated arrays from completed map steps

Context:

Map steps spawn N tasks to process array elements in parallel. This PR ensures their outputs are properly combined back into an array for consumption by downstream steps, completing the map
step infrastructure in the SQL core layer.

changeset-bot · 2025-09-17T07:08:42Z

⚠️ No Changeset found

Latest commit: 8ec0268

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

coderabbitai · 2025-09-17T07:08:45Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

✨ Finishing touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch 09-17-add-map-step-output-aggregation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

jumski · 2025-09-17T07:09:06Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

nx-cloud · 2025-09-17T07:09:32Z

🤖 Nx Cloud AI Fix Eligible

An automatically generated fix could have helped fix failing tasks for this run, but Self-healing CI is disabled for this workspace. Visit workspace settings to enable it and get automatic fixes in future runs.

_{To disable these notifications, a workspace admin can disable them in workspace settings.}

View your CI Pipeline Execution ↗ for commit 8ec0268

Command	Status	Duration	Result
`nx affected -t lint typecheck test --parallel -...`	❌ Failed	6m 6s	View ↗

☁️ Nx Cloud last updated this comment at 2025-10-05 19:28:30 UTC

pkgs/core/supabase/migrations/20250917115352_pgflow_temp_handle_map_output_aggregation.sql

github-actions · 2025-09-18T09:21:08Z

🔍 Preview Deployment: Website

✅ Deployment successful!

🔗 Preview URL: https://pr-217.pgflow.pages.dev

📝 Details:

Branch: 09-17-add-map-step-output-aggregation
Commit: 49044803ef5e56eb40d9250564be324f06b46eea
View Logs

_Last updated: _

github-actions · 2025-09-18T09:22:53Z

🔍 Preview Deployment: Playground

✅ Deployment successful!

🔗 Preview URL: https://pr-217--pgflow-demo.netlify.app

📝 Details:

Branch: 09-17-add-map-step-output-aggregation
Commit: 49044803ef5e56eb40d9250564be324f06b46eea
View Logs

_Last updated: _

Reflects ongoing work on map step output aggregation, related migration, testing, and documentation efforts, along with current implementation status and pending tasks for map step support in the project.

jumski mentioned this pull request Sep 17, 2025

feat: enhance start_tasks with conditional input logic and add array handling migration #216

Open

jumski temporarily deployed to preview September 17, 2025 07:09 — with GitHub Actions Inactive

jumski changed the title ~~chore: update PLAN.md with current feature and task statuses~~ feat: add Map Step Output Aggregation Sep 17, 2025

jumski marked this pull request as ready for review September 17, 2025 07:12

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 1cc39ef to 93cd2af Compare September 17, 2025 10:54

jumski temporarily deployed to preview September 17, 2025 11:01 — with GitHub Actions Inactive

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 93cd2af to a61429b Compare September 17, 2025 12:07

graphite-app bot reviewed Sep 17, 2025

View reviewed changes

pkgs/core/supabase/migrations/20250917115352_pgflow_temp_handle_map_output_aggregation.sql Show resolved Hide resolved

jumski temporarily deployed to preview September 17, 2025 12:14 — with GitHub Actions Inactive

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from a61429b to b01eb63 Compare September 17, 2025 13:10

jumski temporarily deployed to preview September 17, 2025 14:41 — with GitHub Actions Inactive

jumski force-pushed the 09-17-add-map-step-output-aggregation branch 3 times, most recently from 5a0ef17 to 9ab5b0e Compare September 17, 2025 19:16

jumski temporarily deployed to preview September 17, 2025 19:23 — with GitHub Actions Inactive

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from d02a76c to a3647dc Compare September 17, 2025 19:35

jumski temporarily deployed to preview September 17, 2025 19:43 — with GitHub Actions Inactive

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from a3647dc to 290913f Compare September 17, 2025 20:15

jumski temporarily deployed to preview September 17, 2025 20:23 — with GitHub Actions Inactive

jumski force-pushed the 09-17-add-map-step-output-aggregation branch 2 times, most recently from e6b99ed to b77f743 Compare September 18, 2025 04:35

jumski temporarily deployed to preview September 18, 2025 04:43 — with GitHub Actions Inactive

jumski mentioned this pull request Sep 18, 2025

feat: Add .map() Method to Flow DSL for Map Step Support #218

Open

6 tasks

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from b77f743 to 2b1f97e Compare September 18, 2025 07:32

jumski mentioned this pull request Sep 18, 2025

chore: improve verify-schemas-synced and regenerate-temp-migration #219

Open

jumski temporarily deployed to preview September 18, 2025 07:39 — with GitHub Actions Inactive

jumski force-pushed the 09-17-add-map-step-output-aggregation branch 2 times, most recently from fcb8222 to 7422f2a Compare September 18, 2025 08:46

jumski temporarily deployed to preview September 18, 2025 09:20 — with GitHub Actions Inactive

This was referenced Sep 18, 2025

Fix Orphaned Messages on Run Failure #220

Open

feat: add task_index attribute to step_task_record and start_tasks function #221

Open

test: add integration tests for flows with map steps #222

Open

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 7422f2a to c1c4c34 Compare October 5, 2025 18:59

jumski force-pushed the 09-16-handle-arrays-in-start-tasks branch from 781e413 to 53ca34b Compare October 5, 2025 18:59

jumski mentioned this pull request Oct 5, 2025

docs: improve documentation and READMEs to explain map/array steps #228

Open

chore: update PLAN.md with current feature and task statuses

8ec0268

Reflects ongoing work on map step output aggregation, related migration, testing, and documentation efforts, along with current implementation status and pending tasks for map step support in the project.

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from c1c4c34 to 8ec0268 Compare October 5, 2025 19:20

jumski force-pushed the 09-16-handle-arrays-in-start-tasks branch from 53ca34b to 99d14b5 Compare October 5, 2025 19:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add Map Step Output Aggregation #217

feat: add Map Step Output Aggregation #217

Uh oh!

jumski commented Sep 17, 2025 •

edited

Loading

Uh oh!

changeset-bot bot commented Sep 17, 2025 •

edited

Loading

Uh oh!

coderabbitai bot commented Sep 17, 2025 •

edited

Loading

Review skipped

Other AI code review bot(s) detected

Uh oh!

jumski commented Sep 17, 2025 •

edited

Loading

Uh oh!

nx-cloud bot commented Sep 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

github-actions bot commented Sep 18, 2025

Uh oh!

github-actions bot commented Sep 18, 2025

Uh oh!

Uh oh!

feat: add Map Step Output Aggregation #217

Are you sure you want to change the base?

feat: add Map Step Output Aggregation #217

Uh oh!

Conversation

jumski commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

coderabbitai bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Other AI code review bot(s) detected

Uh oh!

jumski commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nx-cloud bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Nx Cloud AI Fix Eligible

Uh oh!

Uh oh!

github-actions bot commented Sep 18, 2025

🔍 Preview Deployment: Website

Uh oh!

github-actions bot commented Sep 18, 2025

🔍 Preview Deployment: Playground

Uh oh!

Uh oh!

jumski commented Sep 17, 2025 •

edited

Loading

changeset-bot bot commented Sep 17, 2025 •

edited

Loading

coderabbitai bot commented Sep 17, 2025 •

edited

Loading

jumski commented Sep 17, 2025 •

edited

Loading

nx-cloud bot commented Sep 17, 2025 •

edited

Loading