kubernetes ssh not connecting between cc-rs and cc-galera-ss #19
Description
temporarily removing the mysql volme (see issue here #18) i am able to get rs to start, but the applications aren't able to talk to each other via ssh.
CC gives me the following error after the 3 galera nodes come up:
libssh auth error: Access denied. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password (root, key=/root/.ssh/id_rsa)
Inside the galara nodes, here is the contents of authorized_keys:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /keys/cc.pub was not found on this server.</p>
</body></html>
Here are the logs from a galera node:
AUTO_DEPLOYMENT is undefined. Using the default value: 1
CMON_PASSWORD is undefined. Using the default password:
Linking exists. Using CC_HOST=10.51.250.78
Retrieving public key from ClusterControl container
Setting up authorized_keys
============================
Loaded environment variables
============================
HOSTNAME : cc-galera-2
CONTAINER_IP : 10.48.3.39
CLUSTER_TYPE : galera
CLUSTER_NAME : Galera_Docker
VENDOR : mariadb
PROVIDER_VERSION : 10.1
INITIAL_CLUSTER_SIZE : 3
AUTO_DEPLOYMENT : 1
Registering this container with ClusterControl, 10.51.250.78 for automatic deployment
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
dd:54:27:41:95:ee:4b:23:ad:10:14:c4:8e:2c:49:df root@cc-galera-0
The key's randomart image is:
+--[ RSA 2048]----+
+-----------------+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
55:88:23:2d:d4:47:6f:6e:14:84:88:1b:6b:14:0e:c1 root@cc-galera-0
The key's randomart image is:
+--[ DSA 1024]----+
+-----------------+
>> Starting SSHD in the background
And here are the logs from CC:
2018-01-10 04:44:13,470 INFO supervisord started with pid 610
2018-01-10 04:44:14,472 INFO spawned: 'cc-auto-deployment' with pid 613
2018-01-10 04:44:14,474 INFO spawned: 'httpd' with pid 614
2018-01-10 04:44:14,476 INFO spawned: 'sshd' with pid 615
2018-01-10 04:44:14,477 INFO spawned: 'cmon' with pid 616
2018-01-10 04:44:14,479 INFO spawned: 'cmon-ssh' with pid 619
2018-01-10 04:44:14,480 INFO spawned: 'cmon-events' with pid 620
2018-01-10 04:44:14,482 INFO spawned: 'cmon-cloud' with pid 621
2018-01-10 04:44:14,483 INFO spawned: 'mysqld' with pid 622
2018-01-10 04:44:15,813 INFO success: cc-auto-deployment entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-10 04:44:15,813 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-10 04:44:15,813 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-10 04:44:15,813 INFO success: cmon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-10 04:44:15,814 INFO success: cmon-ssh entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-10 04:44:15,814 INFO success: cmon-events entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-10 04:44:15,814 INFO success: cmon-cloud entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-10 04:44:15,814 INFO success: mysqld entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-10 04:44:29,501 INFO exited: cmon (exit status 0; expected)
2018-01-10 04:44:30,504 INFO spawned: 'cmon' with pid 1152
2018-01-10 04:44:31,854 INFO success: cmon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
>> Found the following cluster(s) is yet to deploy:
Galera_Docker
>> Number of containers for Galera_Docker is lower than its initial size (3).
>> Nothing to do. Will check again on the next loop.
>> Found the following cluster(s) is yet to deploy:
Galera_Docker
>> Number of containers for Galera_Docker is lower than its initial size (3).
>> Nothing to do. Will check again on the next loop.
>> Found the following cluster(s) is yet to deploy:
Galera_Docker
>> Found a new set of containers awaiting for deployment. Sending deployment command to CMON.
>> Cluster name : Galera_Docker
>> Cluster type : galera
>> Vendor : mariadb
>> Provider Version : 10.1