[LIVY-702]: Submit Spark apps to Kubernetes #451

askhatri · 2024-06-25T11:05:28Z

This pull request (PR) is the foundational PR for adding Kubernetes support in Apache Livy, originally found here. This update includes a newer version of the Kubernetes client and adds code to display the Spark UI.

Summary of the Proposed Changes

This PR introduces a method to submit Spark applications to a Kubernetes cluster. The key points covered include:

Submitting batch sessions
Submitting interactive sessions
Monitoring sessions, collecting logs, and gathering diagnostic information
Restoring session monitoring after restarts
Garbage collection (GC) of created Kubernetes resources

JIRA link: https://issues.apache.org/jira/browse/LIVY-702

How was this patch tested?

Unit Tests: The patch has been verified through comprehensive unit tests.
Manual Testing: Conducted manual testing using Kubernetes on Docker Desktop.
- Environment: Helm charts.

For detailed instructions on testing using Helm charts, please refer to the documentation available at livycluster.

gyogal · 2024-06-27T16:51:46Z

Thanks for working on this @askhatri and for retesting and updating the original code in PR #249 . If there are no objections, we could merge this PR and any smaller updates or fixes could be added in follow-up tickets before the 0.9 release.

vikas-saxena02 · 2024-07-02T08:30:10Z

can this chnaged be merged if there are not objections?

vikas-saxena02 · 2024-07-09T06:28:11Z

@gyogal @lmccay , can you please take a look at this PR?

jshmchenxi · 2024-07-09T08:09:01Z

@askhatri Thanks for pushing this feature forward!

As a side note, could you please add the original author, @jahstreet, to the author list? We've been using this feature for a year and truly appreciate Alex's initial effort in bringing it to Livy.

askhatri · 2024-07-09T08:57:36Z

Hi @jshmchenxi , I have added the original author, @jahstreet, to the author list as suggested by you. Please let me know incase if any further change or correction required

jahstreet · 2024-07-09T09:10:25Z

Hi @jshmchenxi , I have added the original author, @jahstreet, to the author list as suggested by you. Please let me know incase if any further change or correction required

Thx for mentioning mate, appreciate the credits 🙏 .

jahstreet · 2024-07-09T09:13:38Z

server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala

+  // When istio-proxy restarts, the access to K8s API from livy could be down
+  // until envoy comes back, which could take upto 30 seconds


Is istio/envoy or any ingress controller a prerequisite to using Livy with K8s?

istio/envoy is optional for using Livy with K8s. It is just to collected the Livy logs for audit purpose.

great functionality, this will surely be beneficial

jahstreet · 2024-07-09T09:18:10Z

server/src/main/scala/org/apache/livy/LivyConf.scala

+  val KUBERNETES_GRAFANA_LOKI_ENABLED = Entry("livy.server.kubernetes.grafana.loki.enabled", false)
+  val KUBERNETES_GRAFANA_URL = Entry("livy.server.kubernetes.grafana.url", "http://localhost:3000")
+  val KUBERNETES_GRAFANA_LOKI_DATASOURCE =
+    Entry("livy.server.kubernetes.grafana.loki.datasource", "loki")
+  val KUBERNETES_GRAFANA_TIME_RANGE = Entry("livy.server.kubernetes.grafana.timeRange", "6h")


Great to add some docs about expectations to Grafana and Loki installation to make use of this configs.
Same about installing Livy on K8s, we should give some guide to the community on how to set this up at least locally to play with. That will help to raise the adoption.
I can help you with connecting the dots, ping me if you wanna discuss.

Ahh, I see it in https://github.com/askhatri/livycluster . Great at least to leave somewhere a link to it.

Sure, I will try to add this as part of LIVY-979. I will connect with you via email.

@askhatri better to discuss it on LIVY-979 itself so that whole community is across it.

Sure @vikas-saxena02.

jahstreet · 2024-07-09T09:19:02Z

server/src/main/scala/org/apache/livy/LivyConf.scala

+  // side car container for spark pods enabled?
+  val KUBERNETES_SPARK_SIDECAR_ENABLED =
+    Entry("livy.server.kubernetes.spark.sidecar.enabled", true)


What is the use case for this flag?

A sidecar container in Kubernetes is a secondary container that runs alongside the main application container in the same pod. We can set the sidecar configuration in Livy using this flag.

what could be the intention for running sidecar?

A sidecar container can be used to collect logs from the Spark application and forward them to a centralized logging system like Elasticsearch, Fluentd, and Kibana (EFK) stack.

Understood, thx for explaining. Does Livy add this sidecar or should Livy be handling the Spark Pods with a sidecar differently? What would happen if Spark Pods have sidecar but this flag is false and how it is different from setting it to true? Just trying to understand it better.

Livy does not add a sidecar container for Spark. Instead, Livy uses a flag to determine the status of the Spark pod. If the Spark pod is running with a sidecar, the flag is set to true; otherwise, it is set to false.

Co-authored-by: Alex Sasnouskikh <jahstreetlove@gmail.com>

vikas-saxena02 · 2024-07-09T09:49:45Z

@askhatri the latest push was forced and I can see many of the checks have failed, can you please look into this?

askhatri · 2024-07-09T09:52:37Z

Hi @vikas-saxena02, I have re-triggered the checks now by adding ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true.

vikas-saxena02 · 2024-07-09T11:22:12Z

@jahstreet @askhatri is there any documentation that you can share that explains how to use livy run jobs on spark on kubernetes?

vikas-saxena02 · 2024-07-09T11:22:25Z

/approve

jahstreet · 2024-07-09T13:39:50Z

@jahstreet @askhatri is there any documentation that you can share that explains how to use livy run jobs on spark on kubernetes?

Yes, we will talk about packaging it.
Until then, people can refer:

(newer linked to this PR @askhatri 's setup) https://github.com/askhatri/livycluster
(older my setup) https://github.com/JahstreetOrg/spark-on-kubernetes-helm
Jira issue: https://issues.apache.org/jira/browse/LIVY-979

vikas-saxena02 · 2024-07-09T17:53:40Z

/assign @gyogal

askhatri · 2024-07-10T05:25:14Z

Hi @vikas-saxena02, I have assigned to @gyogal as suggested by you...!

vikas-saxena02 · 2024-07-10T05:57:32Z

Thanks @askhatri !! Hopefully this should be merged soon.

gyogal · 2024-07-10T08:32:13Z

Thanks everyone for your input! It seems like there are no objections and the overall feedback is positive. If you find any issues once this PR is merged, please feel free to raise a ticket.

This pull request (PR) is the foundational PR for adding Kubernetes support in Apache Livy, originally found here (apache#249). This update includes a newer version of the Kubernetes client and adds code to display the Spark UI. ## Summary of the Proposed Changes This PR introduces a method to submit Spark applications to a Kubernetes cluster. The key points covered include: * Submitting batch sessions * Submitting interactive sessions * Monitoring sessions, collecting logs, and gathering diagnostic information * Restoring session monitoring after restarts * Garbage collection (GC) of created Kubernetes resources JIRA link: https://issues.apache.org/jira/browse/LIVY-702 ## How was this patch tested? * Unit Tests: The patch has been verified through comprehensive unit tests. * Manual Testing: Conducted manual testing using Kubernetes on Docker Desktop. * Environment: Helm charts. For detailed instructions on testing using Helm charts, please refer to the documentation available at https://github.com/askhatri/livycluster Co-authored-by: Asif Khatri <asif.khatri@cloudera.com> Co-authored-by: Alex Sasnouskikh <jahstreetlove@gmail.com>

askhatri requested review from gyogal and lmccay June 25, 2024 11:34

askhatri mentioned this pull request Jun 25, 2024

[LIVY-702]: Submit Spark apps to Kubernetes #249

Open

askhatri force-pushed the LIVY-702 branch from 456b965 to d334f64 Compare June 28, 2024 06:02

askhatri force-pushed the LIVY-702 branch from d334f64 to fdbd619 Compare July 9, 2024 08:34

jahstreet reviewed Jul 9, 2024

View reviewed changes

askhatri force-pushed the LIVY-702 branch from 803bad8 to fdbd619 Compare July 9, 2024 09:27

LIVY-702: Submit Spark apps to Kubernetes

8efecc7

Co-authored-by: Alex Sasnouskikh <jahstreetlove@gmail.com>

askhatri force-pushed the LIVY-702 branch from fdbd619 to 8efecc7 Compare July 9, 2024 09:51

askhatri assigned gyogal Jul 10, 2024

gyogal merged commit b089dd6 into apache:master Jul 10, 2024
3 checks passed

askhatri deleted the LIVY-702 branch July 14, 2024 13:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[LIVY-702]: Submit Spark apps to Kubernetes #451

[LIVY-702]: Submit Spark apps to Kubernetes #451

askhatri commented Jun 25, 2024

gyogal commented Jun 27, 2024

vikas-saxena02 commented Jul 2, 2024

vikas-saxena02 commented Jul 9, 2024

jshmchenxi commented Jul 9, 2024

askhatri commented Jul 9, 2024

jahstreet commented Jul 9, 2024

jahstreet Jul 9, 2024

askhatri Jul 9, 2024

vikas-saxena02 Jul 9, 2024

jahstreet Jul 9, 2024

jahstreet Jul 9, 2024

askhatri Jul 9, 2024

vikas-saxena02 Jul 9, 2024

askhatri Jul 9, 2024

jahstreet Jul 9, 2024

askhatri Jul 9, 2024

vikas-saxena02 Jul 9, 2024

askhatri Jul 9, 2024

jahstreet Jul 9, 2024 •

edited

Loading

askhatri Jul 9, 2024

vikas-saxena02 commented Jul 9, 2024

askhatri commented Jul 9, 2024

vikas-saxena02 commented Jul 9, 2024

vikas-saxena02 commented Jul 9, 2024

jahstreet commented Jul 9, 2024 •

edited

Loading

vikas-saxena02 commented Jul 9, 2024 •

edited

Loading

askhatri commented Jul 10, 2024

vikas-saxena02 commented Jul 10, 2024

gyogal commented Jul 10, 2024

		// When istio-proxy restarts, the access to K8s API from livy could be down
		// until envoy comes back, which could take upto 30 seconds

[LIVY-702]: Submit Spark apps to Kubernetes #451

[LIVY-702]: Submit Spark apps to Kubernetes #451

Conversation

askhatri commented Jun 25, 2024

Summary of the Proposed Changes

How was this patch tested?

gyogal commented Jun 27, 2024

vikas-saxena02 commented Jul 2, 2024

vikas-saxena02 commented Jul 9, 2024

jshmchenxi commented Jul 9, 2024

askhatri commented Jul 9, 2024

jahstreet commented Jul 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jahstreet Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vikas-saxena02 commented Jul 9, 2024

askhatri commented Jul 9, 2024

vikas-saxena02 commented Jul 9, 2024

vikas-saxena02 commented Jul 9, 2024

jahstreet commented Jul 9, 2024 • edited Loading

vikas-saxena02 commented Jul 9, 2024 • edited Loading

askhatri commented Jul 10, 2024

vikas-saxena02 commented Jul 10, 2024

gyogal commented Jul 10, 2024

jahstreet Jul 9, 2024 •

edited

Loading

jahstreet commented Jul 9, 2024 •

edited

Loading

vikas-saxena02 commented Jul 9, 2024 •

edited

Loading