Add personalized configuration parameters for each metastore. #315

yangyuxia · 2024-04-14T12:44:15Z

📝 Description

Different hive Metastores may have different parameter values for the same parameter. If multiple federated Metastores are configured in waggle dance, you can configure their own parameter values for each metastore.
For example, when hivemetastore enabled the kerberos authentication, the value of the "hive.metastore.kerberos.principal", some metastore is configured to "hive/_HOST@Realm", but some metastore is configured as "hive/{clusterName}@realm", so you need to configure personalized configuration parameters for each metastore.

🔗 Related Issues

yangyuxia · 2024-04-15T02:00:16Z

cat waggle-dance-server.yml

verbose: true
port: 48869
database-resolution: PREFIXED
yaml-storage:
overwrite-config-on-shutdown: false
logging:
config: file:${WAGGLE_DANCE_HOME}/conf/log4j2.xml
configuration-properties:
hadoop.security.authentication: KERBEROS
hadoop.kerberos.keytab.login.autorenewal.enabled: true
hadoop.proxyuser.hive.users: ''
hadoop.proxyuser.hive.groups: ''
hadoop.proxyuser.hive.hosts: '*'
hive.metastore.sasl.enabled: true
hive.metastore.kerberos.principal: hive/_HOST@HADOOP.COM
hive.metastore.kerberos.keytab.file: ${WAGGLE_DANCE_HOME}/conf/wd.keytab
hive.cluster.delegation.token.store.class: org.apache.hadoop.hive.thrift.ZooKeeperTokenStore
hive.cluster.delegation.token.store.zookeeper.connectString: host-zk1:2181,host-zk2:2181,host-zk3:2181
hive.cluster.delegation.token.store.zookeeper.znode: /hive/cluster/delegationwg
hive.cluster.delegation.token.store.zookeeper.acl: "world:anyone:rwcda"
hive.server2.authentication: KERBEROS
hive.server2.authentication.kerberos.keytab: ${WAGGLE_DANCE_HOME}/conf/wd.keytab
hive.server2.authentication.kerberos.principal: hive/_HOST@HADOOP.COM
hive.server2.authentication.client.kerberos.principal: hive/_HOST@HADOOP.COM
wd-metastore:
  hms1:
    hive.metastore.kerberos.principal: hive/_HOST@HADOOP.COM
  hms2:
    hive.metastore.kerberos.principal: hive/test2@HADOOP.COM

patduin · 2024-04-16T11:49:14Z

...n/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactory.java

@@ -45,7 +43,21 @@ public class CloseableThriftHiveMetastoreIfaceClientFactory {
  public CloseableThriftHiveMetastoreIface newInstance(AbstractMetaStore metaStore) {
    Map<String, String> properties = new HashMap<>();
    if (waggleDanceConfiguration.getConfigurationProperties() != null) {
-      properties.putAll(waggleDanceConfiguration.getConfigurationProperties());
+//      properties.putAll(waggleDanceConfiguration.getConfigurationProperties());
+      Map<String, String> serverConfigMap=waggleDanceConfiguration.getConfigurationProperties();


I suggest we move these properties to the AbstractMetastore class and make it part of the waggle-dance-federation.yml and this code can become much simpler like:

properties.putAll(waggleDanceConfiguration.getConfigurationProperties()); //override per metastore properties.putAll(metastore.getConfigurationProperties());

patduin

Thank you please have a look and my comment.

yangyuxia · 2024-04-22T06:29:44Z

primary-meta-store:
  database-prefix: ''
  name: hms1
  remote-meta-store-uris: thrift://host-hms1:9083
  access-control-type: READ_AND_WRITE_AND_CREATE
  impersonation-enabled: false
  configuration-properties:
    hive.metastore.kerberos.principal: hive/hms@HADOOP.COM
    hive.server2.enable.doAs: true

patduin · 2024-05-06T07:41:26Z

...n/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactory.java

@@ -47,6 +47,8 @@ public CloseableThriftHiveMetastoreIface newInstance(AbstractMetaStore metaStore
    if (waggleDanceConfiguration.getConfigurationProperties() != null) {
      properties.putAll(waggleDanceConfiguration.getConfigurationProperties());
    }
+    //override per metastore
+    properties.putAll(metaStore.getConfigurationProperties());


Could you add a junit test please?

patduin · 2024-05-21T14:58:30Z

waggle-dance-api/src/main/java/com/hotels/bdp/waggledance/api/model/AbstractMetaStore.java

  private @NotBlank String name;
  private @NotBlank String remoteMetaStoreUris;
  private @Valid MetastoreTunnel metastoreTunnel;
  private @NotNull AccessControlType accessControlType = AccessControlType.READ_ONLY;
  private transient @JsonProperty @NotNull MetaStoreStatus status = MetaStoreStatus.UNKNOWN;
  private long latency = 0;
  private transient @JsonIgnore HashBiMap<String, String> databaseNameBiMapping = HashBiMap.create();
-  private Map<String, String> configurationProperties = Collections.emptyMap();
+  private Map<String, String> configurationProperties;// = Collections.emptyMap();


isn't it nicer to instantiate as emptyMap so you don't need the null check?

@yangyuxia if you could revert or explain why this change was made I'm happy to approver once that's done,thanks!

patduin · 2024-05-21T14:58:57Z

waggle-dance-api/src/main/java/com/hotels/bdp/waggledance/api/model/AbstractMetaStore.java

@@ -51,15 +51,15 @@ public abstract class AbstractMetaStore {
  private List<String> writableDatabaseWhitelist;
  private List<String> mappedDatabases;
  private @Valid List<MappedTables> mappedTables;
-  private Map<String, String> databaseNameMapping = Collections.emptyMap();
+  private transient Map<String, String> databaseNameMapping = Collections.emptyMap();


why this had to change?

@yangyuxia if you could revert or explain why this change was made I'm happy to approver once that's done,thanks!

jmnunezizu · 2024-05-22T17:31:03Z

waggle-dance-api/src/main/java/com/hotels/bdp/waggledance/api/model/AbstractMetaStore.java

@@ -243,4 +244,12 @@ public String toString() {
        .toString();
  }

+  public Map<String, String> getConfigurationProperties() {


Please move these two methods below the setDatabaseNameMapping method to keep them in sync with the property definitions. Normally, toString (if defined) is the last method in a class.

jmnunezizu · 2024-05-22T17:32:56Z

...n/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactory.java

@@ -47,6 +47,10 @@ public CloseableThriftHiveMetastoreIface newInstance(AbstractMetaStore metaStore
    if (waggleDanceConfiguration.getConfigurationProperties() != null) {
      properties.putAll(waggleDanceConfiguration.getConfigurationProperties());
    }
+    //override per metastore


The comment isn't needed.
nit: there's a space missing between if and (. You might also want to leave an empty line before the return statement to improve readability.

jmnunezizu · 2024-05-22T17:34:14Z

...va/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactoryTest.java

-
-    factory.newInstance(newFederatedInstance("fed1", THRIFT_URI));
+    FederatedMetaStore fed1 = newFederatedInstance("fed1", THRIFT_URI);
+    fed1.setConfigurationProperties(Collections.singletonMap(ConfVars.METASTORE_KERBEROS_PRINCIPAL.varname,"hive/_HOST@HADOOP.COM"));


Nit: there's a space missing between the key and the value.

jmnunezizu · 2024-05-22T17:37:28Z

Thanks for the contribution. Please update the readme explaining how this works and how it can be utilised. Thanks.

jmnunezizu

Thanks!

patduin

Thank you

using new HashMap so the generated Yaml doesn't generate an anchor (reference &id001)

fixing test

…aGroup#315) * Add personalized configuration parameters for each metastore. * Add personalized configuration parameters for each metastore * Recover * Update junit test * Update Junit Test * Update Junit Test * Update Junit Test * Format the code and update the readme * Revert * Update FederatedMetaStoreTest.java * Update PrimaryMetaStoreTest.java * Update AbstractMetaStore.java using new HashMap so the generated Yaml doesn't generate an anchor (reference &id001) * Update YamlFederatedMetaStoreStorageTest.java fixing test --------- Co-authored-by: yangyx <360508847@qq.com> Co-authored-by: Patrick Duin <patduin@gmail.com>

…e delegatetoken in the kerberos environment (#313) * Use different tokens instead of forcing WD and all HMS to use the same delegatetoken in the kerberos environment * Add personalized configuration parameters for each metastore. (#315)

Add personalized configuration parameters for each metastore.

d94d0a9

patduin reviewed Apr 16, 2024

View reviewed changes

patduin requested changes Apr 16, 2024

View reviewed changes

yangyx and others added 3 commits April 19, 2024 17:19

Add personalized configuration parameters for each metastore

5b43c27

Merge branch 'ExpediaGroup:hive-3.x' into hive-3.x

e9a93a1

Recover

394250d

patduin reviewed May 6, 2024

View reviewed changes

yangyx added 4 commits May 11, 2024 14:43

Update junit test

14abc77

Update Junit Test

877a76a

Update Junit Test

085d7fc

Update Junit Test

cb7b7bf

patduin reviewed May 21, 2024

View reviewed changes

jmnunezizu added enhancement hive3 labels May 22, 2024

jmnunezizu requested changes May 22, 2024

View reviewed changes

Format the code and update the readme

8307404

jmnunezizu approved these changes May 24, 2024

View reviewed changes

Revert

f837b2e

patduin approved these changes May 26, 2024

View reviewed changes

patduin added 4 commits May 27, 2024 09:48

Update FederatedMetaStoreTest.java

bc64c92

Update PrimaryMetaStoreTest.java

8cb4b22

Update AbstractMetaStore.java

5bd959c

using new HashMap so the generated Yaml doesn't generate an anchor (reference &id001)

Update YamlFederatedMetaStoreStorageTest.java

23258d2

fixing test

patduin merged commit 842aa2e into ExpediaGroup:hive-3.x May 27, 2024
0 of 2 checks passed

flaming-archer mentioned this pull request Jul 2, 2024

Can we implement a MetaStoreFilterHook to modify the path and support HMS that uses the same ns #320

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add personalized configuration parameters for each metastore. #315

Add personalized configuration parameters for each metastore. #315

yangyuxia commented Apr 14, 2024

yangyuxia commented Apr 15, 2024

patduin Apr 16, 2024

patduin left a comment

yangyuxia commented Apr 22, 2024 •

edited

Loading

patduin May 6, 2024

yangyuxia May 11, 2024

patduin May 21, 2024

patduin May 24, 2024

patduin May 21, 2024

patduin May 24, 2024

jmnunezizu May 22, 2024

jmnunezizu May 22, 2024

jmnunezizu May 22, 2024

jmnunezizu commented May 22, 2024

jmnunezizu left a comment

patduin left a comment

Add personalized configuration parameters for each metastore. #315

Add personalized configuration parameters for each metastore. #315

Conversation

yangyuxia commented Apr 14, 2024

📝 Description

🔗 Related Issues

yangyuxia commented Apr 15, 2024

Choose a reason for hiding this comment

patduin left a comment

Choose a reason for hiding this comment

yangyuxia commented Apr 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmnunezizu commented May 22, 2024

jmnunezizu left a comment

Choose a reason for hiding this comment

patduin left a comment

Choose a reason for hiding this comment

yangyuxia commented Apr 22, 2024 •

edited

Loading