Skip to content

# 🚀 Task Spawning for Map Steps #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: 9-11-root-map-support
Choose a base branch
from
Open

# 🚀 Task Spawning for Map Steps #213

wants to merge 1 commit into from

Conversation

jumski
Copy link
Contributor

@jumski jumski commented Sep 12, 2025 •
edited
Loading

Summary

Implements task spawning functionality for map steps, enabling parallel task creation based on array input length. This PR enhances start_ready_steps() to generate N tasks for map steps while maintaining backward compatibility with single steps.

What's New

  • ✅ Dynamic Task Spawning: Map steps spawn N tasks based on initial_tasks count
  • ✅ Empty Array Handling: Zero-length arrays auto-complete with [] output
  • ✅ Batch Operations: Efficient task creation using generate_series()
  • ✅ Task Indexing: Proper task_index assignment (0 to N-1) for ordered execution

Implementation Details

  • Enhanced start_ready_steps() to handle both single and map step types
  • Added CROSS JOIN LATERAL generate_series() for efficient batch task creation
  • Special handling for empty arrays - direct transition to completed state
  • Maintains existing behavior for single steps (1 task with index 0)

Testing

  • Comprehensive test coverage for task spawning scenarios
  • Tests for empty array auto-completion
  • Validation of task_index values and message queue entries
  • Performance testing script for large arrays

Next Steps

This completes the task spawning phase. Next PRs will implement:

  • Array element extraction in start_tasks()
  • Dependent map support in complete_task()
  • Output aggregation in maybe_complete_run()

Part of the map infrastructure implementation (PR #4 of 8)

@changeset-bot changeset-bot
Copy link

changeset-bot bot commented Sep 12, 2025 •
edited
Loading

⚠️ No Changeset found

Latest commit: f462cf0

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 12, 2025 •
edited
Loading

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

✨ Finishing touches
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch 09-12-task-spawning

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@jumski jumski mentioned this pull request Sep 12, 2025
Open
@jumski Graphite App
Copy link
Contributor Author

jumski commented Sep 12, 2025 •
edited
Loading

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

This was referenced Sep 12, 2025
Open
Open
@nx-cloud Nx Cloud
Copy link

nx-cloud bot commented Sep 12, 2025 •
edited
Loading

View your CI Pipeline Execution ↗ for commit f462cf0

Command Status Duration Result
nx run-many -t build --projects client,dsl --co... ✅ Succeeded 4s View ↗
nx affected -t build --configuration=production... ✅ Succeeded 3s View ↗
nx affected -t lint typecheck test --parallel -... ✅ Succeeded 5m 55s View ↗

☁️ Nx Cloud last updated this comment at 2025-09-15 18:08:59 UTC

@jumski jumski changed the title feat: Add scripts and tests for task spawning and step execution in flow management # 🚀 Task Spawning for Map Steps Sep 12, 2025
@jumski jumski marked this pull request as ready for review September 12, 2025 12:44
@jumski jumski force-pushed the 09-12-task-spawning branch 2 times, most recently from 9bd624e to 66e3c3b Compare September 12, 2025 13:24
@jumski jumski force-pushed the 09-12-task-spawning branch 2 times, most recently from ce696e8 to 850de93 Compare September 12, 2025 15:09
@jumski jumski force-pushed the 09-12-task-spawning branch from 850de93 to 05e2388 Compare September 12, 2025 21:37
graphite-app[bot]
graphite-app bot reviewed Sep 12, 2025
View reviewed changes
@jumski jumski mentioned this pull request Sep 15, 2025
Open
@jumski
feat: Add scripts and tests for task spawning and step execution in f…
f462cf0 
…low management

- Introduced collect_perf_data.sh for performance testing of large array handling
- Updated start_ready_steps function to handle empty map steps and initialize task states
- Added migration script to modify start_ready_steps for correct task spawning
- Created tests for map step message queueing, delayed message scheduling, and task spawning
- Ensured proper handling of initial_tasks, task indices, and step status transitions
- Included tests for both map and single steps to verify correct task creation and message
dispatching
@jumski jumski force-pushed the 09-12-task-spawning branch from 05e2388 to f462cf0 Compare September 15, 2025 17:54
@jumski jumski force-pushed the 9-11-root-map-support branch from c0fc2a1 to 2e5052c Compare September 15, 2025 17:54
@github-actions GitHub Actions
Copy link
Contributor

🔍 Preview Deployment: Website

✅ Deployment successful!

🔗 Preview URL: https://pr-213.pgflow.pages.dev

📝 Details:

  • Branch: 09-12-task-spawning
  • Commit: b954636eae34c888374155dd6a533f213569ad6f
  • View Logs

_Last updated: _

@github-actions GitHub Actions
Copy link
Contributor

🔍 Preview Deployment: Playground

✅ Deployment successful!

🔗 Preview URL: https://pr-213--pgflow-demo.netlify.app

📝 Details:

  • Branch: 09-12-task-spawning
  • Commit: b954636eae34c888374155dd6a533f213569ad6f
  • View Logs

_Last updated: _

@jumski jumski mentioned this pull request Sep 16, 2025
Open
This was referenced Sep 16, 2025
Open
Open
Open
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@graphite-app graphite-app[bot] graphite-app[bot] left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jumski