Document building SWT natives from command line

[trancexpress]

The https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/Readme.md points to: https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/Build-SWT-native-binaries-for-running-platform.launch

Previously there was a build script (that still exists, but no idea how to use it) that could be ran for the rebuild. The README described how to call this script.

Please document how to build SWT natives from command line too.

[trancexpress]

Hi @HannesWell ,

could you take a look here?

Also, how do I actually compile SWT natives from command line? I've tried different things, the build.sh script doesn't find make files regardless of where I try to trigger the script from and what ENV variables I pass to it.

Do you have an idea, what options do we have to recompile SWT natives during the Eclipse mvn build (we only need to recompile the Linux 64 bit GTK3 binaries)? We ran into #1631 and we hope to avoid the problem (for now) by just recompiling SWT native binaries. So far no idea how to do this (resulted in this ticket) to test, but ideally we trigger the compilation during the build - as opposed to mvn downloading the binaries from the internet.

Best regards and thanks,
Simeon

[iloveeclipse]

Note: we've built SWT natives before (latest with 4.30) from command line locally, and that worked (after SWT native source code changes) via

cd eclipse.platform.releng.aggregator/eclipse.platform.swt.binaries/bundles/org.eclipse.swt.gtk.linux.x86_64
mvn clean verify -Pbuild-individual-bundles -Dnative=gtk.linux.x86_64

cd eclipse.platform.releng.aggregator/eclipse.platform.swt/bundles/org.eclipse.swt
mvn clean verify -Pbuild-individual-bundles

So we need instructions how to perform something similar that doesn't need m2e / UI at all with the "new" SWT build in 4.35.

[HannesWell]

Also, how do I actually compile SWT natives from command line?

The short answer is that you can do what is done in the vicinity of

eclipse.platform.swt/Jenkinsfile

Line 220 in 2ece1af

runOnNativeBuildAgent("${PLATFORM}") {

or is done by the mentioned launch-config respectively in the binaries/pom.xml.

I can look into providing more explicit instructions for CLI builds in the next days, but IIRC the last time the doc was enhanced we agreed that just looking at the build scripts avoids documentation that might become outdated respectively is not correctly maintained.

[trancexpress]

Also, how do I actually compile SWT natives from command line?

The short answer is that you can do what is done in the vicinity of

eclipse.platform.swt/Jenkinsfile

Line 220 in 2ece1af

runOnNativeBuildAgent("${PLATFORM}") {

or is done by the mentioned launch-config respectively in the binaries/pom.xml.

I can look into providing more explicit instructions for CLI builds in the next days, but IIRC the last time the doc was enhanced we agreed that just looking at the build scripts avoids documentation that might become outdated respectively is not correctly maintained.

When I try to run that script (as mentioned above) there are errors:

$:/data/git/eclipse.platform.releng.aggregator/eclipse.platform.swt/bundles/org.eclipse.swt/Eclipse SWT PI/gtk/library[st_4x]$ MODEL=x86_64 OUTPUT_DIR=/data/git/eclipse.platform.releng.aggregator/eclipse.platform.swt/binaries/org.eclipse.swt.gtk.linux.x86_64 SWT_JAVA_HOME=/usr/lib/jvm/java-17/ sh build.sh install
Cairo found, compiling SWT support for the cairo graphics library.
jni.h found
libjawt.so found, the SWT/AWT integration library will be compiled.
Building SWT/GTK+ for Architectures: linux x86_64
==== Building GTK3 + GTK 4 ====
Note: When building multiple GTK versions, a cleanup is required (and automatically performed) between them.
Cleaning up...
make_linux.mak:31: make_common.mak: No such file or directory
make: *** No rule to make target `make_common.mak'.  Stop.
Building GTK4 bindings:
make_linux.mak:31: make_common.mak: No such file or directory
make: *** No rule to make target `make_common.mak'.  Stop.
*** GTK4 Build failed, aborting further actions..

I don't know what the working directory of the Jenkins task is?

After copying everything from the common directory, to the Linux directory, the make complains about a missing "c.o" target, that seems to be present in the Windows and MacOS build files, but not in the Linux one.

The Maven for Eclipse launch does work fine (I see the expected GTK+ compile outputs), aside from trying to compile for GTK4 when there is no GTK4 installed. It would be nice to know what this launch calls on command line, so that we can call it from command line.

[trancexpress]

Alright, I see. There is a program that collect sources to a folder, from that folder the build.sh script can be run.

Thanks @HannesWell for pointing me to the Jenkins file!

I'll document my findings once we have something running; generally we are interested in rebuilding SWT natives from scratch whenever we build the SDK (for our target OS and architecture).

[laeubi]

So we need instructions how to perform something similar that doesn't need m2e / UI at all with the "new" SWT build in 4.35.

What exactly is not working with that? e.g. we also have the github actions that compile the natives as well as part of the build:

https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/.github/workflows/maven.yml

if you use -DskipTests as an option it should also only compile and skip the test execution entirely.

[trancexpress]

So we need instructions how to perform something similar that doesn't need m2e / UI at all with the "new" SWT build in 4.35.

What exactly is not working with that? e.g. we also have the github actions that compile the natives as well as part of the build:

https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/.github/workflows/maven.yml

if you use -DskipTests as an option it should also only compile and skip the test execution entirely.

I tried the -Dnative option, it does seem to compile natives. Any way to limit the compile to GTK3? We don't have GTK4 in one of our supported environments.

[HannesWell]

The Maven for Eclipse launch does work fine (I see the expected GTK+ compile outputs), aside from trying to compile for GTK4 when there is no GTK4 installed. It would be nice to know what this launch calls on command line, so that we can call it from command line.

Essentially you have to do, what is done in

eclipse.platform.swt/binaries/pom.xml

Lines 123 to 146 in e8452d8

	<property name="build_dir" value="${project.build.directory}/natives-build-temp"/>
	<exec executable="${java.home}/bin/java" dir="${swtMainProject}" failonerror="true">
	<arg line="-Dws=${ws} -Darch=${arch} build-scripts/CollectSources.java -nativeSources '${build_dir}'"/>
	</exec>
	<property name="SWT_JAVA_HOME" value="${java.home}"/><!-- Not overwritten if already set, e.g. as CLI argument -->
	<echo>Compile SWT natives against headers and libraries from JDK: ${SWT_JAVA_HOME}</echo>
	<if><equals arg1="${ws}" arg2="win32" />
	<then>
	<exec dir="${build_dir}" executable="${build_dir}/build.bat" failonerror="true">
	<env key="SWT_JAVA_HOME" value="${SWT_JAVA_HOME}"/>
	<env key="OUTPUT_DIR" value="${project.basedir}"/>
	<arg line="install clean"/>
	</exec>
	</then>
	<else>
	<exec dir="${build_dir}" executable="sh" failonerror="true">
	<arg line="build.sh"/>
	<env key="SWT_JAVA_HOME" value="${SWT_JAVA_HOME}"/>
	<env key="OUTPUT_DIR" value="${project.basedir}"/>
	<env key="MODEL" value="${arch}"/>
	<arg line="install clean"/>
	</exec>
	</else>
	</if>

I.e.

1. Call build-scripts/CollectSources.java
2. Call ${build_dir}/build.sh/bat

with all the required working- and output-directories variables as well as os/ws/arch and java-home (env) variables being set to the corresponding values. What you have to call and set exactly depends on your OS and ARCH, but the snippet above explains it best.

And as it was said, you can also just call the mentioned Build-SWT-native-binaries-for-running-platform launch configuration.
This configuration just calls from within the directory binaries/org.eclipse.swt.${target.ws}.${target.os}.${target.arch} in this repo:
mvn clean antrun:run@build-native-binaries -Dnative=${target.ws}.${target.os}.${target.arch}

Doing this just requires you to set ${target.ws}.${target.os}.${target.arch} to the corresponding values, e.g. win32.win32.x86_64 or gtk.linux.aarch64 and the SWT native binaries will be built against the JDK running the Maven build.
Of course this expects that all build tools are present as listed in

• https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/Readme.Win32.md#building-natives
• https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/Readme.Linux.md#building-64bit-natives-for-gtk
• https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/Readme.macOS.md#building-natives

Just for completes, as you have found it before:
https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/Readme.md#building-and-testing-locally also contains general information, which basically point to the things above.

[trancexpress]

Thank you for all the infos @HannesWell ! I'll see what we can document without too much risk of having outdated info, when I get to it.

Right now we bump the revisions in SWT as done by the Jenkins file, we rename all the SWT binaries that already exist (from git LFS) to the new revision (since we can't compile most of them), then we run the source collection program and then the build.sh for GTK3 only. That, and committing the changes, is enough to have the aggregator build use SWT compiled by us.

[akurtakov]

What (if anything) is planned to be done here?

[trancexpress]

What (if anything) is planned to be done here?

We are still testing if the steps we do actually work; unfortunately its taking a while.

After we are sure the build works as expected, I'll document our steps in the wiki. Then we can close this ticket.

[akurtakov]

Is there a progress here or this one should be closed ?

[trancexpress]

Thank you for reminding me @akurtakov , at this point I think this is enough: https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/Readme.md#building-and-testing-locally

We ended up abandoning our script to rebuild SWT natives, due to the fix here: #1791

The script we ended up temporarily using has quite a few steps... We would likely be creating yet another outdated Wiki page if we would document it; using the mvn launch and target should be enough.