Skip to content

Commit

Permalink
[CELEBORN-1240][FOLLOWUP] Introduce web profile for web module
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Introduce web profile for web module.

### Why are the changes needed?

The compilation speed of web module is sometimes very slow due to the influence of the network, which hinders the development process.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

CI.

Closes #2679 from SteNicholas/CELEBORN-1240.

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>

(cherry picked from commit c0dda4a)
  • Loading branch information
SteNicholas authored and FMX committed Jan 3, 2025
1 parent 139e47f commit 1f01635
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 2 deletions.
8 changes: 7 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@
<module>service</module>
<module>master</module>
<module>worker</module>
<module>web</module>
</modules>

<distributionManagement>
Expand Down Expand Up @@ -1634,6 +1633,13 @@
</modules>
</profile>

<profile>
<id>web</id>
<modules>
<module>web</module>
</modules>
</profile>

<profile>
<id>google-mirror</id>
<properties>
Expand Down
23 changes: 22 additions & 1 deletion project/CelebornBuild.scala
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,7 @@ object CelebornBuild extends sbt.internal.BuildDef {
CelebornClient.client,
CelebornService.service,
CelebornWorker.worker,
CelebornMaster.master) ++ maybeSparkClientModules ++ maybeFlinkClientModules ++ maybeMRClientModules
CelebornMaster.master) ++ maybeSparkClientModules ++ maybeFlinkClientModules ++ maybeMRClientModules ++ maybeWebModules
}

// ThisBuild / parallelExecution := false
Expand Down Expand Up @@ -401,6 +401,15 @@ object Utils {

lazy val maybeMRClientModules: Seq[Project] = mrClientProjects.map(_.modules).getOrElse(Seq.empty)

val WEB_VERSION = profiles.filter(_.startsWith("web")).headOption

lazy val webProjects = WEB_VERSION match {
case Some("web") => Some(WebProjects)
case _ => None
}

lazy val maybeWebModules: Seq[Project] = webProjects.map(_.modules).getOrElse(Seq.empty)

def defaultScalaVersion(): String = {
// 1. Inherit the scala version of the spark project
// 2. if the spark profile not specified, using the DEFAULT_SCALA_VERSION
Expand Down Expand Up @@ -1236,3 +1245,15 @@ object MRClientProjects {
}
)
}

object WebProjects {

def web: Project = {
Project("celeborn-web", file("web"))
.settings(commonSettings)
}

def modules: Seq[Project] = {
Seq(web)
}
}

0 comments on commit 1f01635

Please sign in to comment.