Add optional enable_drain_healthcheck feature

Notes:
This feature cannot be used with the arbitrator. If it is enabled and
the deployment uses the arbitrator, then drain will always fail.

When enabled, this feature will prevent deleting deployments, unless the --force flag
is used.

[#157656403]
Signed-off-by: Caroline Taymor <ctaymor@pivotal.io>

Author: APShirley

jobs/mysql/spec

@@ -83,6 +83,9 @@ properties:
   cf_mysql.mysql.max_connections:
     description: 'Maximum total number of database connections for the node'
     default: 1500
+  cf_mysql.mysql.enable_drain_healthcheck:
+    description: 'When this is enabled, the --skip-drain flag is required in order to delete a deployment. Drain healthcheck cannot be enabled when using the arbitrator. Enabling this ensures mysql nodes will verify the health of all other nodes before draining.'
+    default: false
   cf_mysql.mysql.tls.ca_certificate:
     description: 'CA Certificate which signed the server certificate'
   cf_mysql.mysql.tls.server_certificate:

jobs/mysql/templates/drain.sh

@@ -2,7 +2,7 @@
 
 <% if p('cf_mysql_enabled') == true %>
 set -e -o pipefail
-
+<% if p('cf_mysql.mysql.enable_drain_healthcheck') == true %>
 <%
   cluster_ips = link('mysql').instances.map(&:address)
   if_link('arbitrator') do
@@ -36,7 +36,7 @@ for NODE in "${CLUSTER_NODES[@]}"; do
     exit -1
   fi
 done
-
+<% end %>
 # Actually drain with a kill_and_wait on the mysql pid
 PIDFILE=/var/vcap/sys/run/mariadb_ctl/mariadb_ctl.pid
 source /var/vcap/packages/cf-mysql-common/pid_utils.sh