VMDistributedCluster CR: orchestrate VMCluster upgrades

[vrutkovs]

Add a new CR - VMDistributedCluster - so that multiple VMClusters can be upgraded in an orchestrated fashion, ensuring the read VMAuth is disabled before upgrade and the VMAgent (if available) doesn't have pending bytes to send.

Fixes #1515

This CR can refer to VMClusters using one of two possible ways:

• Existing VMClusters can be referred to using ref property and changes applied using overrideSpec
• Entirely new VMClusters can be created with name and spec properties

Either way, settings in VMDistributedCluster would be applied to target VMClusters, overriding their existing settings if necessary.

Current implementation scope:

• VMDistributedCluster requires a single VMAgent instance and multiple VMUsers to track utilization and disable read access when required
• VMDistributedCluster can create new VMCluster instances when name and spec are specified
• VMDistributedCluster can update existing VMCluster objects when ref and overrideSpec are set
• Before the update, the matching VMUser is updated to remove read access to the selected VMCluster
• VMClusters are updated one by one, waiting for them to change status to "operational" again
• After VMCluster update is complete, we're waiting for VMAgent to flush collected data again by checking its metrics
• VMUser is updated to restore read access to VMCluster
• Process is repeated for all remaining VMClusters

See #1515 (comment) for agreed limitations for v1alpha1 version:

• Single VMAlert is supported
• All objects must belong to the same namespace as VMDistributedCluster
• Referenced VMClusters are not being actively watched for changes, they only get reconciled periodically
• All objects must be referred to by name, label selectors are not supported
• Only VMClusters are supported, VMSingles are deferred for other versions
• Upgrading components other than VMClusters is deferred to future version
• Two delays are tweakable:
• • vmclusterWaitReadyDeadline
• • delay between zone updates
• No additional metric to indicate that the cluster is being upgraded to silence possible alerts

TODO:

• Add changelog entry
• Fix flaking tests
• Set ownerRefs to managed VMClusters
• Description-less CRD should be applied for development only. Rephrase descriptions in existing parts to make them fit for production
• Squash commits
  Keeping original commits for review as its useful to show how the feature was developed
• Update existing documentation to mention VMDistributedCluster and describe its target architecture and existing shortcomings

[AndrewChubatiuk]

initially thought distributed CR is needed for full distributed setup management, but looks like it only performs version upgrade. In this case just curious why we need different CRs for VM, VT and VL?

[vrutkovs]

Yes, so far we're focusing on upgrades - existing CRs provide sufficient flexibility IMO - and we didn't get a request for other actions so far.

In this case just curious why we need different CRs for VM, VT and VL?

VL and VT don't have agents (yet) so their specs would be different. However we can reuse the same approach and probably even some helper functions

[Haleygo]

Yes, so far we're focusing on upgrades - existing CRs provide sufficient flexibility IMO - and we didn't get a request for other actions so far.

I believe users would expect to modify the vmcluster spec value or apply extra flags to the vmclusters.
And since vmclusterSpec.ClusterVersion is optional, users could specify component versions inside vmclusterSpec which overrides the vmclusterSpec.ClusterVersion.

And currently, it seems VMDistributedCluster only covers a limited scenario where resources like vmcluster, vmuser, vmauth are defined and configured as needed.
Could you please provide an example of how to config them to achieve similar topology described in victoria-metrics-distributed chart? I expect VMDistributedCluster to be supported there when released.

[vrutkovs]

I believe users would expect to modify the vmcluster spec value or apply extra flags to the vmclusters.

Yup, setting generic overrideParams would be more flexible and, along with upgrades, would cover other maintenance tasks, i.e., adding replicas or setting flags

[Haleygo]

The feature isn’t limited to vmcluster, it can also be applied to vmsingle.
Later, we can support vmsingle with the same CRD by replacing vmcluster objects under VMDistributedClusterSpec with vmsingle.
What about calling it VMDistributed?

[vrutkovs]

Yes, I think it's fair to extend this to VMSingle too. I think it would be trivial to extend this to VMSingles - by adding VMType to Zones property. We didn't get a request to extend this to VMSingle though, so I'd prefer to focus on VMClusters.

Not quite sure about VMDistributed - its fair to treat multiple zones as a cluster (so VMCluster would be a fantastic name :) ), while VMDistributed doesn't really pinpoint it.

[Haleygo]

We didn't get a request to extend this to VMSingle though, so I'd prefer to focus on VMClusters.

I disagree. The VMDistributedCluster audience is the same as the distributed chart’s audience. We’ve received requests to support vmsingle there and added it, see VictoriaMetrics/helm-charts#2090, VictoriaMetrics/helm-charts#2517.
So the need exists, and I believe we should cover it if there are no technical blockers, which I don’t see.
However, the current implementation cannot be extended to support vmsingle in zones without introducing a breaking change.

I think we can either expand the zones, like

Zones []zone `json:"zones,omitempty"`


type zone struct {
  vmclusterList  []VMClusterRefOrSpec `json:"vmclusterList,omitempty"`
   // allow adding vmsingleList now or later
}

or rename the zones field to something like vmclusterList as suggested here, then vmsingleList can be added without affect vmclusterList.

[vrutkovs]

Wouldn't it be easier to update VMClusterRefOrSpec instead? We may also want to define VMSingles inline.

I agree that its best to have a structure for vmsingles ready now

[vrutkovs]

Oh, that makes us rename Spec into ClusterSpec and introduce SingleSpec, which is ugly - so I like your solution better

[Copilot]

Pull Request Overview

This PR introduces a new Custom Resource VMDistributedCluster (v1alpha1) to orchestrate updates across multiple VMCluster instances, ensuring zero-downtime upgrades by coordinating read access disabling and monitoring VMAgent metrics.

Key Changes:

• Added VMDistributedCluster CRD with support for referencing existing or creating new VMCluster instances
• Implemented orchestrated upgrade logic that disables read access, waits for VMAgent to flush data, and re-enables access sequentially
• Added comprehensive e2e tests and unit tests for the new functionality

Reviewed Changes

Copilot reviewed 43 out of 45 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
api/operator/v1alpha1/vmdistributedcluster_types.go	Defines the VMDistributedCluster CRD types including spec and status structures
internal/controller/operator/vmclusterdistributed_controller.go	Main reconciler implementing the orchestrated upgrade logic
internal/controller/operator/factory/vmdistributedcluster/vmdistributedcluster.go	Core business logic for managing VMCluster upgrades, VMUser modifications, and VMAgent metrics monitoring
test/e2e/vmdistributedcluster_test.go	Comprehensive e2e tests covering creation, updates, pausing, and error scenarios
docs/resources/vmdistributedcluster.md	Documentation explaining the CR, its usage, and current limitations
config/samples/*	Example YAML configurations demonstrating various use cases
config/crd/bases/*	Generated CRD manifests
Multiple client/informer/lister files	Auto-generated client code for the new v1alpha1 API

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[f41gh7]

We cannot use external vmagent, only directly defined spec and manage it on our own

[f41gh7]

As an example, external vmagent may have daemonset enabled or deployment mode - which will lose all the data during restarts. And it doesn't make any sense to use it.

Also, it may have scrapping configured. Which is out of scope for distributed cluster.

[f41gh7]

I think, it's better to perform routing on our own and do not ask user to do it.

Any additional VMUsers could be created externally.

[f41gh7]

Lets keep simple Spec for now. If there will be any kind of requests for Ref, we could add it later if needed.

[vrutkovs]

The ref request was initial, existing users want VMDistributedCluster to manage existing VMClusters more efficiently

[f41gh7]

we must deleted all managed resources.

[vrutkovs]

Good point. I think it can be deferred to a followup versions

[f41gh7]

none of example configs works for me:

{"level":"error","ts":"2025-11-21T11:03:43+01:00","logger":"manager","msg":"Reconciler error","controller":"operator-VMDistributedCluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMDistributedCluster","VMDistributedCluster":{"name":"vmdistributedcluster-sample-override","namespace":"default
"},"namespace":"default","name":"vmdistributedcluster-sample-override","reconcileID":"9294af1c-fde0-4d3d-ae4c-c6a5817b9638","error":"parsing object error for object controller=\"vmdistributedcluster\": \"cannot parse vmdistributedcluster spec: {\\\"vmagent\\\":{\\\"name\\\":\\\"vmagent-sample\\\"},\\\"vmusers\\\":[{\
\\"name\\\":\\\"vmuser-us-east-1\\\"},{\\\"name\\\":\\\"vmuser-us-west-2\\\"},{\\\"name\\\":\\\"vmuser-eu-west-1\\\"}],\\\"zones\\\":{\\\"vmclusters\\\":[{\\\"overrideSpec\\\":{\\\"vmselect\\\":{\\\"replicaCount\\\":3}},\\\"ref\\\":\\\"vmcluster-us-east-1\\\"},{\\\"overrideSpec\\\":{\\\"vmselect\\\":{\\\"replicaCount
\\\":4}},\\\"ref\\\":\\\"vmcluster-us-west-2\\\"},{\\\"overrideSpec\\\":{\\\"vmselect\\\":{\\\"replicaCount\\\":2}},\\\"ref\\\":\\\"vmcluster-eu-west-1\\\"}]}}, err: json: cannot unmarshal string into Go struct field VMClusterRefOrSpec.zones.vmclusters.ref of type v1.LocalObjectReference\""}

[f41gh7]

It's not working, see comment above

[f41gh7]

Lets stick with betav1. It's the same as alpha in terms of breaking changes, but it doesn't require any additional entities.

Alpha is another level of abstraction just for abstraction.

[f41gh7]

Lets move this check into Validate method

[f41gh7]

It's better to add panic, it must be impossible case.