-
Notifications
You must be signed in to change notification settings - Fork 602
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[LIVY-702]: Submit Spark apps to Kubernetes #451
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -258,6 +258,63 @@ object LivyConf { | |
// value specifies max attempts to retry when safe mode is ON in hdfs filesystem | ||
val HDFS_SAFE_MODE_MAX_RETRY_ATTEMPTS = Entry("livy.server.hdfs.safe-mode.max.retry.attempts", 12) | ||
|
||
// Kubernetes oauth token file path. | ||
val KUBERNETES_OAUTH_TOKEN_FILE = Entry("livy.server.kubernetes.oauthTokenFile", "") | ||
// Kubernetes oauth token string value. | ||
val KUBERNETES_OAUTH_TOKEN_VALUE = Entry("livy.server.kubernetes.oauthTokenValue", "") | ||
// Kubernetes CA cert file path. | ||
val KUBERNETES_CA_CERT_FILE = Entry("livy.server.kubernetes.caCertFile", "") | ||
// Kubernetes client key file path. | ||
val KUBERNETES_CLIENT_KEY_FILE = Entry("livy.server.kubernetes.clientKeyFile", "") | ||
// Kubernetes client cert file path. | ||
val KUBERNETES_CLIENT_CERT_FILE = Entry("livy.server.kubernetes.clientCertFile", "") | ||
|
||
// If Livy can't find the Kubernetes app within this time, consider it lost. | ||
val KUBERNETES_APP_LOOKUP_TIMEOUT = Entry("livy.server.kubernetes.app-lookup-timeout", "600s") | ||
// How often Livy polls Kubernetes to refresh Kubernetes app state. | ||
val KUBERNETES_POLL_INTERVAL = Entry("livy.server.kubernetes.poll-interval", "15s") | ||
|
||
// How long to check livy session leakage. | ||
val KUBERNETES_APP_LEAKAGE_CHECK_TIMEOUT = | ||
Entry("livy.server.kubernetes.app-leakage.check-timeout", "600s") | ||
// How often to check livy session leakage. | ||
val KUBERNETES_APP_LEAKAGE_CHECK_INTERVAL = | ||
Entry("livy.server.kubernetes.app-leakage.check-interval", "60s") | ||
|
||
// Weather to create Kubernetes Nginx Ingress for Spark UI. | ||
val KUBERNETES_INGRESS_CREATE = Entry("livy.server.kubernetes.ingress.create", false) | ||
// Kubernetes Ingress class name. | ||
val KUBERNETES_INGRESS_CLASS_NAME = Entry("livy.server.kubernetes.ingress.className", "") | ||
// Kubernetes Nginx Ingress protocol. | ||
val KUBERNETES_INGRESS_PROTOCOL = Entry("livy.server.kubernetes.ingress.protocol", "http") | ||
// Kubernetes Nginx Ingress host. | ||
val KUBERNETES_INGRESS_HOST = Entry("livy.server.kubernetes.ingress.host", "localhost") | ||
// Kubernetes Nginx Ingress additional configuration snippet. | ||
val KUBERNETES_INGRESS_ADDITIONAL_CONF_SNIPPET = | ||
Entry("livy.server.kubernetes.ingress.additionalConfSnippet", "") | ||
// Kubernetes Nginx Ingress additional annotations: key1=value1;key2=value2;... . | ||
val KUBERNETES_INGRESS_ADDITIONAL_ANNOTATIONS = | ||
Entry("livy.server.kubernetes.ingress.additionalAnnotations", "") | ||
// Kubernetes secret name for Nginx Ingress TLS. | ||
// Is omitted if 'livy.server.kubernetes.ingress.protocol' value doesn't end with 's' | ||
val KUBERNETES_INGRESS_TLS_SECRET_NAME = | ||
Entry("livy.server.kubernetes.ingress.tls.secretName", "spark-cluster-tls") | ||
|
||
val KUBERNETES_GRAFANA_LOKI_ENABLED = Entry("livy.server.kubernetes.grafana.loki.enabled", false) | ||
val KUBERNETES_GRAFANA_URL = Entry("livy.server.kubernetes.grafana.url", "http://localhost:3000") | ||
val KUBERNETES_GRAFANA_LOKI_DATASOURCE = | ||
Entry("livy.server.kubernetes.grafana.loki.datasource", "loki") | ||
val KUBERNETES_GRAFANA_TIME_RANGE = Entry("livy.server.kubernetes.grafana.timeRange", "6h") | ||
|
||
// side car container for spark pods enabled? | ||
val KUBERNETES_SPARK_SIDECAR_ENABLED = | ||
Entry("livy.server.kubernetes.spark.sidecar.enabled", true) | ||
Comment on lines
+309
to
+311
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is the use case for this flag? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A sidecar container in Kubernetes is a secondary container that runs alongside the main application container in the same pod. We can set the sidecar configuration in Livy using this flag. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what could be the intention for running sidecar? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A sidecar container can be used to collect logs from the Spark application and forward them to a centralized logging system like Elasticsearch, Fluentd, and Kibana (EFK) stack. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Understood, thx for explaining. Does Livy add this sidecar or should Livy be handling the Spark Pods with a sidecar differently? What would happen if Spark Pods have sidecar but this flag is false and how it is different from setting it to true? Just trying to understand it better. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Livy does not add a sidecar container for Spark. Instead, Livy uses a flag to determine the status of the Spark pod. If the Spark pod is running with a sidecar, the flag is set to true; otherwise, it is set to false. |
||
// container name to identify spark pod if running with sidecar containers | ||
val KUBERNETES_SPARK_CONTAINER_NAME = | ||
Entry("livy.server.kubernetes.spark.container.name", "spark-container") | ||
|
||
val UI_HISTORY_SERVER_URL = Entry("livy.ui.history-server-url", "http://spark-history-server") | ||
|
||
// Whether session timeout should be checked, by default it will be checked, which means inactive | ||
// session will be stopped after "livy.server.session.timeout" | ||
val SESSION_TIMEOUT_CHECK = Entry("livy.server.session.timeout-check", true) | ||
|
@@ -371,6 +428,9 @@ class LivyConf(loadDefaults: Boolean) extends ClientConf[LivyConf](null) { | |
/** Return true if spark master starts with yarn. */ | ||
def isRunningOnYarn(): Boolean = sparkMaster().startsWith("yarn") | ||
|
||
/** Return true if spark master starts with k8s. */ | ||
def isRunningOnKubernetes(): Boolean = sparkMaster().startsWith("k8s") | ||
|
||
/** Return the spark deploy mode Livy sessions should use. */ | ||
def sparkDeployMode(): Option[String] = Option(get(LIVY_SPARK_DEPLOY_MODE)).filterNot(_.isEmpty) | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great to add some docs about expectations to Grafana and Loki installation to make use of this configs.
Same about installing Livy on K8s, we should give some guide to the community on how to set this up at least locally to play with. That will help to raise the adoption.
I can help you with connecting the dots, ping me if you wanna discuss.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh, I see it in https://github.com/askhatri/livycluster . Great at least to leave somewhere a link to it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, I will try to add this as part of LIVY-979. I will connect with you via email.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@askhatri better to discuss it on LIVY-979 itself so that whole community is across it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure @vikas-saxena02.