Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BIGTOP-3907: Use self compiled leveldbjni dependency for Spark on aarch64 #1086

Open
wants to merge 1 commit into
base: branch-3.2
Choose a base branch
from
Open

BIGTOP-3907: Use self compiled leveldbjni dependency for Spark on aarch64 #1086

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions bigtop-packages/src/common/spark/do-component-build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,68 @@ set -ex

. `dirname $0`/bigtop.bom

# BIGTOP-3907: Build native library/jars for aarch64
if [ $HOSTTYPE = "aarch64" ] ; then
# download leveldbjni 1.8 from original repo
git clone https://github.com/chirino/leveldb.git
git clone https://github.com/fusesource/leveldbjni.git --branch leveldbjni-1.8 --single-branch
export SNAPPY_HOME=/usr/lib
export LEVELDB_HOME=`cd leveldb; pwd`
export LEVELDBJNI_HOME=`cd leveldbjni; pwd`
export LIBRARY_PATH=${SNAPPY_HOME}
cd ${LEVELDB_HOME}
export C_INCLUDE_PATH=${LIBRARY_PATH}
export CPLUS_INCLUDE_PATH=${LIBRARY_PATH}
# apply ARM64 specific patch
patch -p1 -E << 'EOF'
diff a/port/atomic_pointer.h b/port/atomic_pointer.h
--- a/port/atomic_pointer.h
+++ b/port/atomic_pointer.h
@@ -36,6 +36,8 @@
#define ARCH_CPU_X86_FAMILY 1
#elif defined(__ARMEL__)
#define ARCH_CPU_ARM_FAMILY 1
+#elif defined(__aarch64__)
+#define ARCH_CPU_ARM64_FAMILY 1
#elif defined(__ppc__) || defined(__powerpc__) || defined(__powerpc64__)
#define ARCH_CPU_PPC_FAMILY 1
#endif
@@ -93,6 +95,13 @@ inline void MemoryBarrier() {
}
#define LEVELDB_HAVE_MEMORY_BARRIER

+// ARM64
+#elif defined(ARCH_CPU_ARM64_FAMILY) && defined(__linux__)
+inline void MemoryBarrier() {
+ asm volatile("dmb sy" : : : "memory");
+}
+#define LEVELDB_HAVE_MEMORY_BARRIER
+
// PPC
#elif defined(ARCH_CPU_PPC_FAMILY) && defined(__GNUC__)
inline void MemoryBarrier() {
@@ -216,6 +225,7 @@ class AtomicPointer {
#undef LEVELDB_HAVE_MEMORY_BARRIER
#undef ARCH_CPU_X86_FAMILY
#undef ARCH_CPU_ARM_FAMILY
+#undef ARCH_CPU_ARM64_FAMILY
#undef ARCH_CPU_PPC_FAMILY

} // namespace port
EOF
git apply ../leveldbjni/leveldb.patch
make libleveldb.a
# Now use maven to build and update the local maven repository with
# aarch64 version of leveldbjni.
cd ${LEVELDBJNI_HOME}
mvn clean install -DskipTests -P download -Plinux64,all
cd ..
#cleanup
rm -rf ${LEVELDBJNI_HOME}
rm -rf ${LEVELDB_HOME}
fi
## BIGTOP-3907

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These can be removed if we already built Hadoop

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This issue was brought about by the leveldbjni dependency as you mentioned.
You mean if we build Hadoop first on the same node, it appears there are no leveldbjni dependency issues with Spark, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but we still need to delete profile aarch64 for Spark

BUILD_OPTS="-Divy.home=${HOME}/.ivy2 -Dsbt.ivy.home=${HOME}/.ivy2 -Duser.home=${HOME} \
-Drepo.maven.org=$IVY_MIRROR_PROP \
-Dreactor.repo=file://${HOME}/.m2/repository \
Expand Down
24 changes: 24 additions & 0 deletions bigtop-packages/src/common/spark/patch1-remove-aarch64-profile.diff
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
diff --git a/pom.xml b/pom.xml
index 183d258840..0a5df7944d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -3506,19 +3506,6 @@
<profile>
<id>sparkr</id>
</profile>
- <!-- use org.openlabtesting.leveldbjni on aarch64 platform -->
- <profile>
- <id>aarch64</id>
- <properties>
- <leveldbjni.group>org.openlabtesting.leveldbjni</leveldbjni.group>
- </properties>
- <activation>
- <os>
- <family>linux</family>
- <arch>aarch64</arch>
- </os>
- </activation>
- </profile>
<profile>
<id>jdwp-test-debug</id>
<properties>