Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Spark] Shade the build for Delta Connect projects #3665

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

[Spark] Shade the build for Delta Connect projects #3665

Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5a0f70e
Initial shading
longvu-db Sep 10, 2024
a83c23e
Include back imports
longvu-db Sep 10, 2024
4f1a585
Nit
longvu-db Sep 10, 2024
2411427
Merge branch 'master' of github.com:delta-io/delta into shade-delta-c…
longvu-db Sep 11, 2024
d7be464
Update shade rules
longvu-db Sep 13, 2024
e6bc664
Nit
longvu-db Sep 13, 2024
2110d4f
runOnlySparkMaster assembly
longvu-db Sep 18, 2024
d59fc8d
refactor shading rules
longvu-db Sep 21, 2024
a330575
Nit
longvu-db Sep 21, 2024
211da1f
Trigger Build
longvu-db Sep 22, 2024
713edef
publishM2 only for Spark Master
longvu-db Sep 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 48 additions & 3 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import java.io.BufferedInputStream
import java.nio.file.Files
import java.nio.file.attribute.PosixFilePermission
import java.util
import java.util.Locale

import org.apache.commons.compress.archivers.tar.TarArchiveInputStream
import org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream
Expand Down Expand Up @@ -251,12 +252,14 @@ lazy val connectCommon = (project in file("spark-connect/common"))
task = Test / test,
taskName = "test",
projectName = "delta-connect-common",
emptyValue = ()).value,
emptyValue = ()
).value,
publish := runTaskOnlyOnSparkMaster(
task = publish,
taskName = "publish",
projectName = "delta-connect-common",
emptyValue = ()).value,
emptyValue = ()
).value,
libraryDependencies ++= Seq(
"io.grpc" % "protoc-gen-grpc-java" % grpcVersion asProtocPlugin(),
"io.grpc" % "grpc-protobuf" % grpcVersion,
Expand All @@ -271,6 +274,21 @@ lazy val connectCommon = (project in file("spark-connect/common"))
PB.gens.java -> (Compile / sourceManaged).value,
PB.gens.plugin("grpc-java") -> (Compile / sourceManaged).value
),
(assembly / test) := { },
longvu-db marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically just following other examples in this Delta's build.sbt file

delta/build.sbt

Line 47 in 93eef11

// It was necessary to change default_scala_version to scala213 in build.sbt

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This empty block { } indicates we're not testing this correct?
If so please add a // TODO inside the block.

Copy link
Contributor Author

@longvu-db longvu-db Sep 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @scottsand-db, I just have a quick question.

Why did we always do (assembly / test) := { } for other projects in build.sbt, so basically no testing task for assembly? Is it because we don't need to do it (like QA-ing near a Delta Release is good enough)?

60a582b#diff-5634c415cd8c8504fdb973a3ed092300b43c4b8fc1e184f7249eb29a55511f91

@xupefei In case we should consider adding some testing tasks for assembly, I will add a TODO.

(assembly / logLevel) := Level.Info,
// Exclude `scala-library` from assembly.
(assembly / assemblyPackageScala / assembleArtifact) := false,
(assembly / assemblyShadeRules) := Seq(
ShadeRule.rename("io.grpc.**" -> "org.deltaproject.connect.common.io.grpc.@1").inAll,
ShadeRule.rename("com.google.**" -> "org.deltaproject.connect.common.com.google.@1").inAll,
ShadeRule.rename("javax.annotation.**" -> "org.deltaproject.connect.common.javax.annotation.@1").inAll,
),
(assembly / assemblyMergeStrategy) := {
case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") => MergeStrategy.discard
// Drop all proto files that are not needed as artifacts of the build.
case m if m.toLowerCase(Locale.ROOT).endsWith(".proto") => MergeStrategy.discard
case _ => MergeStrategy.first
}
)

lazy val connectClient = (project in file("spark-connect/client"))
Expand Down Expand Up @@ -357,7 +375,21 @@ lazy val connectClient = (project in file("spark-connect/client"))
} else {
dest.get()
}
}.taskValue
}.taskValue,
(assembly / test) := { },
(assembly / logLevel) := Level.Info,
// Exclude `scala-library` from assembly.
(assembly / assemblyPackageScala / assembleArtifact) := false,
(assembly / assemblyShadeRules) := Seq(
ShadeRule.rename("com.google.**" -> "org.deltaproject.connect.client.com.google.@1").inAll,
ShadeRule.rename("org.scalatest.**" -> "org.deltaproject.connect.client.org.scalatest.@1").inAll,
),
(assembly / assemblyMergeStrategy) := {
case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") => MergeStrategy.discard
// Drop all proto files that are not needed as artifacts of the build.
case m if m.toLowerCase(Locale.ROOT).endsWith(".proto") => MergeStrategy.discard
case _ => MergeStrategy.first
}
)

lazy val connectServer = (project in file("spark-connect/server"))
Expand Down Expand Up @@ -402,6 +434,19 @@ lazy val connectServer = (project in file("spark-connect/server"))
"org.apache.spark" %% "spark-hive" % sparkVersion.value % "test" classifier "tests",
"org.apache.spark" %% "spark-connect" % sparkVersion.value % "test" classifier "tests",
),
(assembly / test) := { },
(assembly / logLevel) := Level.Info,
// Exclude `scala-library` from assembly.
(assembly / assemblyPackageScala / assembleArtifact) := false,
(assembly / assemblyShadeRules) := Seq(
ShadeRule.rename("com.google.**" -> "org.deltaproject.connect.server.com.google.@1").inAll,
),
(assembly / assemblyMergeStrategy) := {
case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") => MergeStrategy.discard
// Drop all proto files that are not needed as artifacts of the build.
case m if m.toLowerCase(Locale.ROOT).endsWith(".proto") => MergeStrategy.discard
case _ => MergeStrategy.first
}
)

lazy val spark = (project in file("spark"))
Expand Down
Loading