[Spark] Shade the build for Delta Connect projects #3665
Conversation
longvu-db
changed the title
build.sbt
Outdated
| (assembly / assemblyPackageScala / assembleArtifact) := false, | ||
| (assembly / assemblyShadeRules) := Seq( | ||
| ShadeRule.rename("com.google.**" -> "org.deltaproject.connect.com.google.@1").inAll, | ||
| ShadeRule.rename("io.grpc.**" -> "org.deltaproject.connect.io.grpc.@1").inAll, |
There was a problem hiding this comment.
By inspecting the jar files of the 3 Delta Connect projects after running publishM2, I believe the shading rule are not propagated to dependent projects, so even if only connectCommon uses io.grpc, we still need to add shading rule io.grpc to connectClient since connectClient depends on connectCommon so connectClient includes connectCommon codes and we need to shade those codes too.
| @@ -271,6 +280,23 @@ lazy val connectCommon = (project in file("spark-connect/common")) | |||
| PB.gens.java -> (Compile / sourceManaged).value, | |||
| PB.gens.plugin("grpc-java") -> (Compile / sourceManaged).value | |||
| ), | |||
| Compile / packageBin := assembly.value, | |||
| (assembly / test) := { }, | |||
There was a problem hiding this comment.
Basically just following other examples in this Delta's build.sbt file
Line 47 in 93eef11
There was a problem hiding this comment.
This empty block { } indicates we're not testing this correct?
If so please add a // TODO inside the block.
There was a problem hiding this comment.
Hey @scottsand-db, I just have a quick question.
Why did we always do (assembly / test) := { } for other projects in build.sbt, so basically no testing task for assembly? Is it because we don't need to do it (like QA-ing near a Delta Release is good enough)?
60a582b#diff-5634c415cd8c8504fdb973a3ed092300b43c4b8fc1e184f7249eb29a55511f91
@xupefei In case we should consider adding some testing tasks for assembly, I will add a TODO.
Which Delta project/connector is this regarding?
Description
Shade the build of
connectServer,connectClientandconnectCommonprojects, this is to avoid conflicts on libraries that Delta Connect library users and the Delta Connect library itself may both depend on such asprotobuf,grpc,...This is needed so that we can ship the Delta Connect project once it is finished.
We hope that this also remove the need of include
com.google.protobuf:protobuf-java:3.25.1in the--packagesto use Delta Connect server and client.https://docs.delta.io/latest/delta-spark-connect.html#how-to-start-the-spark-server-with-delta
How was this patch tested?
Run
sbt assemblyon the projectconnectServer,connectClientandconnectCommon, and use thejar tfcommand to manually inspect the content of the jar files.Does this PR introduce any user-facing changes?
No.