Graylog self-address as master or coordinating node not secure

[PastNullInfinity]

The current way to elect master or slave nodes is poorly implemented.

As it stands, the init-container needs an external dependency with kubectl:

charts/charts/graylog/templates/statefulset.yaml

Lines 74 to 79 in b26ec70

	{{- if .Values.graylog.init.kubectlLocation }}
	wget {{ .Values.graylog.init.kubectlLocation }} -O /k8s/kubectl
	{{- else }}
	wget https://storage.googleapis.com/kubernetes-release/release/{{ .Values.graylog.init.kubectlVersion | default .Capabilities.KubeVersion.Version }}/bin/linux/amd64/kubectl -O /k8s/kubectl
	{{- end }}
	chmod +x /k8s/kubectl

Which is then used to query the k8s nodes to understand whether the current StatefulSet is living in a master or worker node:

charts/charts/graylog/templates/configmap.yaml

Lines 152 to 171 in b26ec70

	# Looking for Master IP
	MASTER_IP=`/k8s/kubectl --namespace {{ .Release.Namespace }} get pod -o jsonpath='{range .items[*]}{.metadata.name} {.status.podIP}{"\n"}{end}' -l graylog-role=master --field-selector=status.phase=Running|awk '{print $2}'`
	SELF_IP=`/k8s/kubectl --namespace {{ .Release.Namespace }} get pod $HOSTNAME -o jsonpath='{.status.podIP}'`
	echo "Current master is $MASTER_IP"
	echo "Self IP is $SELF_IP"
	if [[ -z "$MASTER_IP" ]]; then
	echo "Launching $HOSTNAME as master"
	export GRAYLOG_IS_MASTER="true"
	/k8s/kubectl --namespace {{ .Release.Namespace }} label --overwrite pod $HOSTNAME graylog-role="master"
	else
	# When container was recreated or restart, MASTER_IP == SELF_IP, running as master and no need to change label graylog-role="master"
	if [ "$SELF_IP" == "$MASTER_IP" ];then
	export GRAYLOG_IS_MASTER="true"
	else
	# MASTER_IP != SELF_IP, running as coordinating
	echo "Launching $HOSTNAME as coordinating"
	export GRAYLOG_IS_MASTER="false"
	/k8s/kubectl --namespace {{ .Release.Namespace }} label --overwrite pod $HOSTNAME graylog-role="coordinating"
	fi
	fi

This has two main problems:

1. Introduces a dependency on having a way to download kubectl locally (on-prem deployments or bare-metal ones may not have access to the internet)
2. This breaks the k8s abstraction of making sure that workloads do not need to interface with the Kubernetes API.

I propose to abandon this method, and use a more sensible way to signal each StatefulSet what to do (I'm not too familiar with the product, I'm more than happy to work on a helm-side solution)

[KongZ]

I'm agree that this method is not perfectly optimize since the Graylog itself does not have a mechanism to elect a new master by itself. You have to manual choose the master by set it on configuration file before Graylog is started.

Since the product itself does not support Master selection the init-container here introduced the solution by using only helm without modifying product.

Alternate solution would hard-code pod-0 to Master but in some circumstance which pod-0 lost or could not start, we will lost Master and Graylog will stop working.

Other solutions are welcome. You can freely have a discussion here.

And for the on-prem or no internet access, you can set .Values.graylog.init.kubectlLocation to download a kubectl from other locations.