[feat, multicast] Multicast Group+Member Support #9091
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This work introduces multicast IP pool capabilities to support external multicast traffic routing through the rack's switching infrastructure. Includes: - Add IpPoolType enum (unicast/multicast) with unicast as default - Add multicast pool fields: switch_port_uplinks (UUID[]), mvlan (VLAN ID) - Add database migration (multicast-support/up01.sql) with new columns and indexes - Add ASM/SSM range validation for multicast pools to prevent mixing - Add pool type-aware resolution for IP allocation - Add custom deserializer for switch port uplinks with deduplication - Update external API params/views for multicast pool configuration - Add SSM constants (IPV4_SSM_SUBNET, IPV6_SSM_FLAG_FIELD) for validation Database schema updates: - ip_pool table: pool_type, switch_port_uplinks, mvlan columns - Index on pool_type for efficient filtering - Migration preserves existing pools as unicast type by default This provides the foundation for multicast group functionality while maintaining full backward compatibility with existing unicast pools. References (for review): - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14
zeeshanlakhani
changed the base branch from
main
to
zl/ip-pool-multicast-support
zeeshanlakhani
force-pushed
the
zl/mcast-impl
branch
from
7eec6c5 to
889f1ef
Compare
zeeshanlakhani
changed the title
Introduces end-to-end multicast group support across control plane and sled-agent, integrated with IP pool extensions required
for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming
and exposes an API for operating multicast groups over instances.
Highlights:
- DB: new multicast_group tables; member lifecycle management
- API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback
- Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking
- Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests
- Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present
- Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/
Components:
- Database schema: external and underlay multicast groups; member/instance association tables
- Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence
- API layer: endpoints and validation; default-VNI semantics when VPC not provided
- Sled agent: OPTE stubs and compatibility shims for older agents
Workflows Implemented:
1. Instance lifecycle integration:
- "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW
- "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack
- "Stop" -> deactivate dataplane membership; retain DB membership for fast restart
- "Delete" -> remove instance memberships; group deletion is explicit
- "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees
- Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming
2. RPW reconciliation:
- ensure dataplane switches match database state
- handle sled migrations and state transitions
- Eventual consistency with retry logic
Migrations:
- Apply schema changes in schema/crdb/multicast-group-support/up01.sql (and update dbinit.sql)
- Bump schema versions accordingly
API/Compatibility:
- OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json
- Contains a version change (to v5) as InstanceEnsureBody has been modified to
include multicast_groups associated with an instance in the
underlying sled config
- Regenerate clients where applicable
References:
- RFD 488: https://rfd.shared.oxide.computer/rfd/488
- IP Pool extensions: #9084
- Dendrite PRs (based on recency):
* oxidecomputer/dendrite#132
* oxidecomputer/dendrite#109
* oxidecomputer/dendrite#14
Follow-ups include:
- OPTE integration
- commtest extension
- omdb commands are tracked in issues
- pool and group stats
zeeshanlakhani
force-pushed
the
zl/mcast-impl
branch
from
889f1ef to
04dfa49
Compare
zeeshanlakhani
requested review from
FelixMcFelix,
internet-diglett,
rcgoodfellow and
taspelund
…sed on config
Being that we still have OPTE and Maghemite updates to come for statically routed multicast,
we gate RPW and Saga actions behind runtime configuration ("on" for tests). API calls
are tagged "experimental."
zeeshanlakhani
force-pushed
the
zl/mcast-impl
branch
from
8c6215e to
ca242df
Compare
Collaborator
Author
|
@internet-diglett, others, I added "feature-gating" to this PR, as well as "experimental" tagging for the new entrypoints. |
1 task
rcgoodfellow
reviewed
Oct 9, 2025
Contributor
rcgoodfellow
left a comment
rcgoodfellow
left a comment
There was a problem hiding this comment.
A few API questions to start out with.
Removes project-scoping on multicast groups, adds documentation to API paths doing similar logic for members (through groups vs instances)
Includes:
* Documentation cleanup across the board
* Schema+Model
- Remove rack_id from ExternalMulticastGroup model and database schema
* Reconciler -> Backplane Port Resolution + Refactor `handle` fns
- Add sled → switch port mapping cache with TTL
- Fetch backplane map from DPD for topology validation
- Resolve sled_id → SP (via inventory collection call) → sp_slot → rear port
- Validate sp_slot values against hardware backplane map
- Cache mappings per-sled with automatic invalidation on topology
changes
- Refactor member state processing logic
* Dataplane Client
- Add fetch_backplane_map() for topology validation from DPD-client
- Refactor drift detection and better logging
- Extend member add/remove operations with port resolution
* Simulation Infrastructure
- Add FAKE_GIMLET_MODEL constant ("i86pc") in sp-sim
- Update sled-agent-sim to use sp_sim::FAKE_GIMLET_MODEL
- Add for_testing_with_baseboard() helper for custom baseboard configs
- Enables inventory-based sled/SP matching in tests
* Testing
- Add integration_tests/inventory_matching.rs test
- Update multicast tests for inventory-based port resolution
- Add ensure_inventory_ready() helper for RPW reconciler tests
* Config
- nexus_config additions for cache TTLs, etc
Collaborator
Author
|
@rcgoodfellow @internet-diglett @FelixMcFelix ok, this is back up for review. |
Also includes: * rm datastore param from dataplane client (no longer used) * `_`var removal/cleanup
rcgoodfellow
reviewed
Nov 7, 2025
Contributor
rcgoodfellow
left a comment
rcgoodfellow
left a comment
There was a problem hiding this comment.
Thanks @zeeshanlakhani. Changes address my primary concerns. A few questions here on cache invalidation.
Sorry if I missed them, but if they're not here can we add some tests around the cases where we expect cache invalidation to kick in e.g. inventory changes and ttl timeouts?
Let's also sync up with @askfongjojo on taking this on a lap through the product assurance test suite.
Includes:
- Stale port cleanup: When cache invalidation occurs (manual or via
topology changes), reconciler now removes members from old switch ports
before programming new ones. Prevents stale forwarding state.
- We now compute a union of active member ports across all "Joined"
members to safely prune only stale ports
- We also add a fallback removal path for when `sled_id` is
unavailable on the verify path (e.g., member.sled_id is NULL or sled
removed)
- Wired up cache invalidation flag and inventory watchers:
- Adds `AtomicBool` flag shared between reconciler and Nexus for manual
cache invalidation signaling
- Connects inventory collection/load watchers to reconciler to trigger
automatic updates when topology changes
- Reconciler clears invalidation flag after processing
- Adds cache invalidation tests, better error handling, etc
Collaborator
Author
|
@rcgoodfellow, cache updates in. @askfongjojo, lmk when you want to sync up on testing this. I already noted the enablement flag as well (since this is still experimental). |
I deployed the PR to a racklet and ran the same regression tests I did for a R17 release candidate and haven't observed any difference in functional behavior or TCP network I/O perf. The tests cover mainly firewall and VPC custom routes, VM-to-VM VPC subnet throughput, some storage benchmarks (which are more for exercising the propolis-to-crucible network path). On this PR, I did see higher datagram loss % (>10%) under multiple threads, e.g., whereas on rack2, I typically get I'll re-test on another racklet to see if it's because of environment factors. Update (11/10/2025): A re-test on the same racklet using a r17 release candidate yield similar datagram loss rate. So the issue is specific to the racklet environment, not this PR. Sorry about the confusion @zeeshanlakhani. |
rcgoodfellow
approved these changes
Nov 14, 2025
Contributor
rcgoodfellow
left a comment
rcgoodfellow
left a comment
There was a problem hiding this comment.
Thanks @zeeshanlakhani. All my review comments have been addressed. Appreciate all the hard work in getting this initial infrastructure for multicast groups integrated into the control plane.
Collaborator
Author
|
Thanks @rcgoodfellow. @internet-diglett, did you have any follow-ups or anything on the versioning? If not, I'll merge tonight/day. |
zeeshanlakhani
changed the title
zeeshanlakhani
added a commit
to oxidecomputer/oxnet
that referenced
this pull request
Nov 18, 2025
This aligns better with the current Multicast work in Omicron: oxidecomputer/omicron#9091. Includes: * Handle IPv4 Local (and org-local mcast) * Better naming conventions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements end-to-end multicast networking across Omicron's control plane and sled-agent, integrated with IP pool extensions from #9084.
Closes #8242.
TL;DR:
Highlights:
v7with multicast join/leave endpointssled_id→ rear switch-ports (backplane) for instancesmvlan: External groups support an optional Multicast VLAN for (eventual) upstream egressInstanceSledLocalConfig.multicast_groupsAPI Endpoints:
The instance-scoped endpoints provide an alternative interface for the same join/leave operations, and there's also the system-level IP lookup endpoint.
New Sagas:
multicast_group_dpd_ensure: Ties together external/underlay creation of groups on all switchesmulticast_group_dpd_update: Updates group configuration across switchesBreaking Changes:
v6tov7v208→v209)Migration Notes:
References: