Support multiple namespaces

[frouioui]

Context

Over the last few years, we have received several reports (#236 and #8172) from people saying that the vitess-operator does not support multiple namespaces. Meaning, the operator only works when both the operator pod and the Vitess cluster are in the same K8S namespace.

The vitess-operator, like other Kubernetes operators built with the Operator SDK, uses the built-in WATCH_NAMESPACE environment variable to determine which namespaces to observe and manage. According to the Operator SDK documentation (docs), this variable accepts a comma-separated list of namespaces. Note that this detail is not explicitly documented in the vitess-operator documentation or the example scripts we provide, as this behavior is inherited from how Kubernetes operators work.

We know the operator already sees and manages multiple namespaces thanks to WATCH_NAMESPACE, the rest of the code was also written in a way where every resources inside a VitessCluster inherit the same namespace.

Changes

In order to provide more clarity and better examples to our end-users I have modified the examples we provide. They now contain all the configuration required to run with two namespaces: default and example. Where default hosts the vitess-operator pod, and example the entire Vitess cluster. With this change I am also changing our E2E test scripts to create the example namespace and use -n example as an option to kubectl where applicable.

The changes needed to run with multiple namespaces are the following:

• The WATCH_NAMESPACE variable that we set in operator.yaml has to contain a comma-separated list of all the namespaces we want to have. (code)

• The Role named vitess-operator now has to become a ClusterRole in order to not be namespaced and allow the operator to manage resources across the entire K8S cluster. The ServiceAccount named vitess-operator also have to be binded cluster-wide, using a ClusterRoleBinding instead of a RoleBinding.

• The Vitess backup subcontroller, VitessBackupStorage, creates its own fork of the vitess-operator pod, into a new pod located in the same namespace as the VitessCluster. This new operator process only runs a single reconciling loop, a simplified version of the root operator process, which will watch and manage new backups in the VitessCluster. To achieve this, it needs API access to these resources: VitessShards, VitessBackups and VitessBackupsStorages. New RBAC have to be added allowing access to these resources, along with a new ServiceAccount in the target namespace (same namespace as the VitessCluster). The new RBAC and ServiceAccount then have to be binded using a RoleBinding.

Documentation and vitessio/vitess Changes

To avoid conflict and unnecessary work, once #658 has been merged along with its two follow-up PRs: vitessio/website#1946 and vitessio/vitess#17869 - I will work on creating the follow up PRs on the documentation repository and the vitessio/vitess repository to reflect the changes made in this PR.

Related Issues

• Closes vitess-operator not managing/creating clusters in other namespaces #236
• Related to Watch all namespaces vitessio/vitess#8172