[core] Moved namespaces and computes to database #3446
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amitsrivastava
requested review from
wing2fly,
JohanAhlen,
ranade1,
athithyaaselvam and
Harshg999
ranade1
reviewed
Aug 29, 2023
| @@ -0,0 +1,203 @@ | |||
| #!/usr/bin/env python | |||
There was a problem hiding this comment.
how frequently does the sync warehouse happen? Is this script responsible for deleting VW which has gone away?
wing2fly
approved these changes
Aug 31, 2023
wing2fly
reviewed
Aug 31, 2023
Harshg999
reviewed
Sep 5, 2023
|
|
||
| response[interface] = namespaces | ||
| response['status'] = 0 | ||
|
|
||
| return JsonResponse(response) | ||
|
|
||
|
|
||
| @api_error_handler | ||
| def get_context_computes(request, interface): |
desktop/core/src/desktop/management/commands/sync_warehouses.py
Outdated
Show resolved
Hide resolved
| @@ -1800,7 +1800,7 @@ def get_config(self): | |||
| ], | |||
| 'default_sql_interpreter': default_sql_interpreter, | |||
| 'cluster_type': self.cluster_type, | |||
| 'has_computes': self.cluster_type in ('altus', 'snowball'), # or any grouped engine connectors | |||
| 'has_computes': self.cluster_type in ('cdw', 'altus', 'snowball'), # or any grouped engine connectors | |||
There was a problem hiding this comment.
Let's remove altus and snowball? They are not used anywhere right?
@JohanAhlen Any idea?
There was a problem hiding this comment.
I am thinking of creating a separate commit to remove the altus/snowball related code.
amitsrivastava
force-pushed
the
dev/amit/compute-with-namespace-master-aug-25
branch
2 times, most recently
from
f81856f
to
074dae5
Compare
Done the downgrade test. Rollback worked fine. |
amitsrivastava
force-pushed
the
dev/amit/compute-with-namespace-master-aug-25
branch
5 times, most recently
from
7a1c082
to
3ea8782
Compare
amitsrivastava
force-pushed
the
dev/amit/compute-with-namespace-master-aug-25
branch
from
3ea8782
to
7b326ed
Compare
amitsrivastava
force-pushed
the
dev/amit/compute-with-namespace-master-aug-25
branch
3 times, most recently
from
3c9a591
to
acaeb5f
Compare
The existing implementation was connecting to external Altus and DWX api's to fetch the available computes. This commit changes that by moving the cluster config to database. Two new tables have been added (a) `beeswax_namespace` to hold config for a namespace/dialect (b) `beeswax_compute` to hold configs for individual compute clusters linked to the namespaces. This change currently support hive and impala clusters. There is also a service-discovery component that keeps the list of namespaces and computes updated in the corresponding tables. `sync_warehouses.py` performs the service discovery in the CDW environ by talking to kubernetes api's. It creates one Hive and one Impala namespace and one compute for each virtual warehouse. The command is supposed to run every minute and keep the list of warehouses updated. There are related changes and fixes to the rest of the code to support the query execution on different computes. Change-Id: Ifd8dbc8d716dfe2000fbfa8121e39f2610051fa1
amitsrivastava
force-pushed
the
dev/amit/compute-with-namespace-master-aug-25
branch
from
acaeb5f
to
3594a73
Compare
amitsrivastava
deleted the
dev/amit/compute-with-namespace-master-aug-25
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The existing implementation was connecting to external Altus and DWX
api's to fetch the available computes. This commit changes that by
moving the cluster config to database. Two new tables have been added
(a)
beeswax_namespaceto hold config for a namespace/dialect(b)
beeswax_computeto hold configs for individual compute clusterslinked to the namespaces.
This change currently support hive and impala clusters.
There is also a service-discovery component that keeps the list of
namespaces and computes updated in the corresponding tables.
sync_warehouses.pyperforms the service discovery in the CDW environby talking to kubernetes api's. It creates one Hive and one Impala
namespace and one compute for each virtual warehouse. The command is
supposed to run every minute and keep the list of warehouses updated.
There are related changes and fixes to the rest of the code to support
the query execution on different computes.
Change-Id: Ifd8dbc8d716dfe2000fbfa8121e39f2610051fa1