[feat, multicast] Add IP pool multicast support #9084

zeeshanlakhani · 2025-09-25T01:23:42Z

This work introduces multicast IP pool capabilities to support external multicast traffic routing through the rack's switching infrastructure.

Closes #8217.

Includes:

Add IpPoolType enum (unicast/multicast) with unicast as default
Add database migration (multicast-support/up01.sql) with new columns and indexes
Add ASM/SSM range validation for multicast pools to prevent mixing
Add pool type-aware resolution for IP allocation
Add custom deserializer for switch port uplinks with deduplication
Update external API params/views for multicast pool configuration
Add SSM constants (IPV4_SSM_SUBNET, IPV6_SSM_FLAG_FIELD) for validation

Database schema updates:

ip_pool table: pool_type
Index on pool_type for efficient filtering
Migration preserves existing pools as unicast type by default

This provides the foundation for multicast group functionality while maintaining full backward compatibility with existing unicast pools.

References (for review):

RFD 488: https://rfd.shared.oxide.computer/rfd/488
Dendrite PRs (based on recency):

TODOs:

Multicast Group Support ([feat, multicast] Multicast Group+Member Support #9091)

This work introduces multicast IP pool capabilities to support external multicast traffic routing through the rack's switching infrastructure. Includes: - Add IpPoolType enum (unicast/multicast) with unicast as default - Add multicast pool fields: switch_port_uplinks (UUID[]), mvlan (VLAN ID) - Add database migration (multicast-support/up01.sql) with new columns and indexes - Add ASM/SSM range validation for multicast pools to prevent mixing - Add pool type-aware resolution for IP allocation - Add custom deserializer for switch port uplinks with deduplication - Update external API params/views for multicast pool configuration - Add SSM constants (IPV4_SSM_SUBNET, IPV6_SSM_FLAG_FIELD) for validation Database schema updates: - ip_pool table: pool_type, switch_port_uplinks, mvlan columns - Index on pool_type for efficient filtering - Migration preserves existing pools as unicast type by default This provides the foundation for multicast group functionality while maintaining full backward compatibility with existing unicast pools. References (for review): - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14

Cargo.toml

Introduces end-to-end multicast group support across control plane and sled-agent, integrated with IP pool extensions required for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming and exposes an API for operating multicast groups over instances. Highlights: - DB: new multicast_group tables; member lifecycle management - API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback - Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking - Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests - Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present - Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/ Components: - Database schema: external and underlay multicast groups; member/instance association tables - Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence - API layer: endpoints and validation; default-VNI semantics when VPC not provided - Sled agent: OPTE stubs and compatibility shims for older agents Workflows Implemented: 1. Instance lifecycle integration: - "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW - "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack - "Stop" -> deactivate dataplane membership; retain DB membership for fast restart - "Delete" -> remove instance memberships; group deletion is explicit - "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees - Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming 2. RPW reconciliation: - ensure dataplane switches match database state - handle sled migrations and state transitions - Eventual consistency with retry logic Migrations: - Apply schema changes in schema/crdb/multicast-support/up01.sql (and update dbinit.sql) - Bump schema versions accordingly API/Compatibility: - OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json - Regenerate clients where applicable References: - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - IP Pool extensions: #9084 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14 Follow-ups include: - OPTE integration - commtest extension - omdb commands are tracked in issues - pool and group stats

Introduces end-to-end multicast group support across control plane and sled-agent, integrated with IP pool extensions required for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming and exposes an API for operating multicast groups over instances. Highlights: - DB: new multicast_group tables; member lifecycle management - API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback - Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking - Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests - Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present - Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/ Components: - Database schema: external and underlay multicast groups; member/instance association tables - Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence - API layer: endpoints and validation; default-VNI semantics when VPC not provided - Sled agent: OPTE stubs and compatibility shims for older agents Workflows Implemented: 1. Instance lifecycle integration: - "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW - "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack - "Stop" -> deactivate dataplane membership; retain DB membership for fast restart - "Delete" -> remove instance memberships; group deletion is explicit - "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees - Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming 2. RPW reconciliation: - ensure dataplane switches match database state - handle sled migrations and state transitions - Eventual consistency with retry logic Migrations: - Apply schema changes in schema/crdb/multicast-group-support/up01.sql (and update dbinit.sql) - Bump schema versions accordingly API/Compatibility: - OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json - Contains a version change (to v5) as InstanceEnsureBody has been modified to include multicast_groups associated with an instance in the underlying sled config - Regenerate clients where applicable References: - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - IP Pool extensions: #9084 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14 Follow-ups include: - OPTE integration - commtest extension - omdb commands are tracked in issues - pool and group stats

…support

nexus/types/src/external_api/params.rs

…support

… pools

zeeshanlakhani · 2025-10-09T15:13:11Z

~~New changes pushed onto this branch, but the PR is not picking it up: bcb4fc6#diff-6e5a0e9d23e313fccb92594537052a498855670462b49a8748d4ed3611ba4b37 and a merge commit to bring in the schema updates.~~ (it now works)

rcgoodfellow

Thanks for the updates @zeeshanlakhani. Comments follow from a full read through.

nexus/types/src/external_api/params.rs

common/src/address.rs

nexus/db-queries/src/db/datastore/ip_pool.rs

nexus/db-queries/src/db/datastore/switch_port.rs

nexus/src/app/ip_pool.rs

rcgoodfellow

LGTM. @internet-diglett can you take a look in terms of overall omicron integration?

internet-diglett

Great work! There were a few things that stuck out to me where I know you are probably just working around existing code, but it might be worth updating when we / if can do so.

common/src/vlan.rs

nexus/db-model/src/ip_pool.rs

Also, we add DB a constraint and specialize the range error.

…support

zeeshanlakhani · 2025-10-15T05:35:15Z

@internet-diglett made that change for ranges if you wanna 👀. Updating from main as well.

Implements end-to-end multicast networking across Omicron's control plane and sled-agent, integrated with IP pool extensions from #9084. Closes #8242. TL;DR: > Implements fleet-wide multicast groups across the control plane and sled-agent, integrated with IP pool extensions (#9084). Adds a reconciliation worker (RPW), inventory-based sled→switch-port mapping, a multi-switch multicast dataplane trait, and paired external/underlay groups for NAT and Source-Specific Multicast (SSM). Introduces fleet-scoped auth and a 3-state membership lifecycle; requires schema `v209` and sled-agent API `v7`; feature is disabled by default. *Highlights*: - An RPW for reconciling groups and instance members (ensuring dataplane state matches DB) - Inventory-based sled→switch-port mapping with validation tests - A multicast-focused dataplane trait separating control plane logic from Dendrite/DPD; works across multiple switches - Bifurcated architecture with paired external/underlay groups for NAT-based forwarding - 3-state instance member lifecycle ("Joining" → "Joined" → "Left") with reactivation support - Fleet-scoped authorization model allowing cross-project multicast - New DB tables: multicast_group, underlay_multicast_group, multicast_group_member - External groups: Customer-facing IPv4/IPv6 addresses from IP pools with SSM support - Underlay groups: Admin-scoped IPv6 (ff04::/16); default allocation from fixed ff04::/64 for internal rack forwarding - Feature flag and reconciler/cache settings exist and default to disabled/safe values - Member states: "Joining"/"Joined"/"Left" with soft-delete/mark-for-removal for instance lifecycle - Group states: "Creating"/"Active"/"Deleting"/"Deleted" for RPW processing - sled-agent: API `v7` with multicast join/leave endpoints - Multicast-aware instance management - Network interface configuration for multicast traffic - OPTE integration stubbed pending oxidecomputer/opte#847 - Inventory / Port correlation - Validates baseboard identifiers match between sleds and SPs - Required for multicast reconciler to map `sled_id` → rear switch-ports (backplane) for instances - `mvlan`: External groups support an optional Multicast VLAN for (eventual) upstream egress - Updates to instance sagas as Nexus passes memberships to sled-agent via `InstanceSledLocalConfig.multicast_groups` *API Endpoints*: - GET /v1/multicast-groups: List fleet multicast groups - POST /v1/multicast-groups: Create multicast group - GET /v1/multicast-groups/{group}: View group details - PUT /v1/multicast-groups/{group}: Update group (name, sources) - DELETE /v1/multicast-groups/{group}: Delete group - GET /v1/multicast-groups/{group}/members: List group members - POST /v1/multicast-groups/{group}/members: Add instance to group - DELETE /v1/multicast-groups/{group}/members/{instance}: Remove instance from group - GET /v1/instances/{instance}/multicast-groups: List groups for an instance - PUT /v1/instances/{instance}/multicast-groups/{group}: Join instance to group - DELETE /v1/instances/{instance}/multicast-groups/{group}: Leave group - GET /v1/system/multicast-groups/by-ip/{address}: Lookup group by IP address The instance-scoped endpoints provide an alternative interface for the same join/leave operations, and there's also the system-level IP lookup endpoint. *New Sagas*: - `multicast_group_dpd_ensure`: Ties together external/underlay creation of groups on all switches - `multicast_group_dpd_update`: Updates group configuration across switches *Breaking Changes*: - sled-agent API version bump from `v6` to `v7` - New required configuration in Nexus (multicast.enabled flag, reconciler period, and cache TTL settings) - Schema migration required (`v208` → `v209`) *Migration Notes*: - Multicast as a feature is disabled by default for safe rollout - Multicast endpoints are marked as "experimental" *References*: - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - IP Pool extensions: #9084 - Dendrite PRs (based on recency): - oxidecomputer/dendrite#132 - oxidecomputer/dendrite#109 - oxidecomputer/dendrite#14

zeeshanlakhani force-pushed the zl/ip-pool-multicast-support branch from b692148 to 98bf71e Compare September 25, 2025 04:18

zeeshanlakhani requested review from FelixMcFelix, internet-diglett and rcgoodfellow September 25, 2025 06:37

zeeshanlakhani commented Sep 25, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

zeeshanlakhani mentioned this pull request Sep 25, 2025

[feat, multicast] Multicast Group+Member Support #9091

Merged

zeeshanlakhani self-assigned this Sep 25, 2025

zeeshanlakhani requested a review from bnaecker September 25, 2025 16:13

zeeshanlakhani changed the title ~~[feat] Add IP pool multicast support~~ [feat, multicast] Add IP pool multicast support Sep 25, 2025

zeeshanlakhani requested a review from taspelund September 26, 2025 02:19

zeeshanlakhani added 2 commits September 30, 2025 08:13

Merge remote-tracking branch 'origin/main' into zl/ip-pool-multicast-…

6bd8e19

…support

Merge remote-tracking branch 'origin/main' into zl/ip-pool-multicast-…

20db268

…support

rcgoodfellow reviewed Oct 9, 2025

View reviewed changes

nexus/types/src/external_api/params.rs Outdated Show resolved Hide resolved

zeeshanlakhani added 2 commits October 9, 2025 11:16

Merge remote-tracking branch 'origin/main' into zl/ip-pool-multicast-…

6fe0c6a

…support

[review] move mvlan and switch port uplinks (for mcast egress) out of…

bcb4fc6

… pools

zeeshanlakhani force-pushed the zl/ip-pool-multicast-support branch from 0e7ad02 to bcb4fc6 Compare October 9, 2025 15:29

[fmt] fixes

4433066

zeeshanlakhani requested a review from rcgoodfellow October 9, 2025 15:39

rcgoodfellow reviewed Oct 11, 2025

View reviewed changes

zeeshanlakhani added 2 commits October 12, 2025 06:11

[review] comments, validation, & cleanup

f7991c3

[api] run generate post-API changes

1b24936

rcgoodfellow approved these changes Oct 12, 2025

View reviewed changes

internet-diglett approved these changes Oct 13, 2025

View reviewed changes

common/src/vlan.rs Show resolved Hide resolved

nexus/db-model/src/ip_pool.rs Outdated Show resolved Hide resolved

[review] have IP range conversions use tryfrom

666446f

Also, we add DB a constraint and specialize the range error.

zeeshanlakhani requested a review from internet-diglett October 14, 2025 04:23

Merge remote-tracking branch 'origin/main' into zl/ip-pool-multicast-…

c636f66

…support

[fix] db ordering

f52ce73

zeeshanlakhani merged commit 773799e into main Oct 15, 2025
17 checks passed

zeeshanlakhani deleted the zl/ip-pool-multicast-support branch October 15, 2025 13:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[feat, multicast] Add IP pool multicast support #9084

[feat, multicast] Add IP pool multicast support #9084

Uh oh!

zeeshanlakhani commented Sep 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Oct 9, 2025 •

edited

Loading

Uh oh!

rcgoodfellow left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow left a comment

Uh oh!

internet-diglett left a comment

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Oct 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[feat, multicast] Add IP pool multicast support #9084

[feat, multicast] Add IP pool multicast support #9084

Uh oh!

Conversation

zeeshanlakhani commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rcgoodfellow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow left a comment

Choose a reason for hiding this comment

Uh oh!

internet-diglett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Oct 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zeeshanlakhani commented Sep 25, 2025 •

edited

Loading

zeeshanlakhani commented Oct 9, 2025 •

edited

Loading