Support CDP Base 7.1.8 for ECS autodeployment on EC2 #88
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add control keys for autotls, pvc_type, free_ipa to control deployment behavior more simply. * Standardise the host group name for ECS nodes to be 'ecs_nodes' to match the other standard groups we use in Ansible inventory * Deprecate duplicate filter_null_configs filter from api_client * Add handler to restart cloudera management service * Migrate autotls implementation * Migrate cms_tls setup * Add default external_auth configuration to generally handle freeipa or mit setup * Update importAdminCredentials command to not fail when already imported, but report other errors * Add new role cloudera_manager.services_info to perform useful service discovery on existing clusters. * Migrate role to set session_timeout for cloudera_manager * Migrate role to set hue_ticket_lifetime for PvC-DS deployments * Migrate role to setup TLS for KMS * Migrate role to fix some libs for the Oozie UI in some PvC-DS deployments * Migrate role to setup some default Ranger policies for some PvC-DS deployments * Migrate role to setup a SOLR role in Knox for some PvC-DS deployments * Migrate role to ensure a Ranger plugin for SOLR is deployed in some PvC-DS deployments * Update the defaults for database type and version to respond to el7 or el8 appropriately. * Migrate role to setup WXM. * Update krb5_client deployment for FreeIPA setup, including a patch for dbus_session config and specific configs for when running PvC-DS. * Default krb5_domain to krb5_realm.lower automatically. * Add default kerberos configuration to krb5_common, including simple defaults for when MIT KDC or Red Hat IPA are selected. Passwords default to the cloudera_manager_admin_password instead of hardcoded values like 'changeme' * Add fixes for FreeIPA server deployment * Fix refresh_ranger_kms_repo role to function correctly when determining the Ranger URL in modern Ansible. * Add operation to restart a given cluster or a given cluster's services or cluster management services for user convenience. They could be handlers, but this felt more useful as more people know how to use roles than handlers. * Migrate role to setup iptables or nftables for PvC ECS deployment on Rhel7 or Rhel8 * Add firewalld to unwanted services during automated os prereq setup * Add fix where setting up postgresql_connector sometimes requires python3-psycopg2 to be setup for SSB. * Migrate role to set up a subset of necessary local accounts on ecs_nodes * Add check to ensure that FreeIPA and a custom repo are not on the same host as they both try to hardcode port 8443. * Enhance error message when TLS setup is only being partially applied to hosts in the cluster definition * Update ecs cluster template to set version to DATA_SERVICES1 to reflect current Cloudera Manager 7.6.5 requirements * Modifie ecs services Jinja template to seek host groups by long name. * Explicitly set default database_type to postgresql to avoid user confusion * Add nfs-utils to OS prereqs when installing ECS * Add control for whether or not embedded database mode for ECS is implemented * Remove unused deployment.j2 template * Add controlPlaneValuesEmbedded.j2 for embedded database values * Fix bug in services.j2 for ECS deployment where it would look for the wrong host template name * Rename free_ipa switch to freeipa_activated to match other top level switches * Allow Cloudera Manager version and distro to be set explicitly for repo setup * Update default cloudera-manager version to 7.6.5 * Fix custom_repo to recognise ecs_nodes as valid * Update dbus patch for freeipa client to only restart services if something is changed * Add autodns support to freeipa clients * Add autodns function to freeipa server setup, including creating required zones and records for PvC-DS ECS if that is being deployed * Add task to Flush and Delete IPTables when setting up ECS * Set default Cloudera Manager version to 7.6.1 for base deployments. (7.6.5 is primarily for PvC-DS.) * Add draft ECS teardown processes * Add cloudera.cluster.operations.stop_cluster as a convenience method, as ECS needs to be stopped and cleaned in a specific sequence. * Provide additional wildcard DNS records for ECS in FreeIPA Autodns setup Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
Signed-off-by: Webster Mudge <wmudge@cloudera.com>
…dera-labs#83) * Fix download and reuse of Cloudera Manager repo-as-tarball * Switch to using custom_repo as base url for archive.cloudera.com by default if custom_repo is specified in build without a specific base_url being supplied. * Fix kts setup for RHEL8+ where gpg 2.1+ is used which has changed the default file set - look for kbx files. * Setup of automatic DNS on the freeipa server now supports running on ec2 instances with RHEL8.6, and EL8 generally * Automatically set selinux to permissive on the krb5_server for RHEL8, as otherwise setup is blocked * Separate Py2 and Py3 setup on RHEL8 in preparation for final Py2 deprecation in Cloudera products * Set default admins group as configurable variable in freeipa config * Add option to ensure creation of a 'superuser' in FreeIPA as the default 'admin' user can clash with system users within CDP in some circumstances. This user is then useful to PvC-DS installs and not created by default otherwise. Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
… policy differentiator for solr ranger policy in 7.1.8 during PvC deployment Handled idempotence failure when applying ranger policies Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
Adding cloudera repo management as dependency for deployment services and external auth setup as it contains the defaults for CM and CDH versions. Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ranger policy objects have changed in 7.1.8 from 7.1.7
Added default policy differentiator for solr ranger policy in 7.1.8 during PvC deployment
Handled idempotence failure when applying ranger policies
Signed-off-by: Daniel Chaffelson chaffelson@gmail.com