Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
buildbox: Add new cross-compiling buildbox for Teleport (#44130)
* buildbox: Add new cross-compiling buildbox for Teleport
Add Dockerfiles and make targets to build a new buildbox that has a set
of cross compilers for the four architectures we target for Linux -
amd64, arm64, 386, and arm (32-bit). The buildbox also contains the
third-party C libraries that Teleport needs to statically link against
for a full set of features, again compiled for each target architecture.
The cross compilers are built using crosstool-NG. The compilers must
have a "vendor" field in the "target triple" of "unknown" (default).
Setting it to anything else means rust does not find the cross compilers
properly. Using "unknown" causes the triple to match the rust target.
(This may ultimately be unnecessary to make these match, but is needed
if we have the boring crate build boringssl itself - see last
paragraph).
Two container images are produced:
* `buildbox-thirdparty` - a base image that contains the third-party
tools (compilers, etc) and libraries that we build from source. Once
build, this should never need to be rebuilt unless we change the
version of one of the components it builds.
* `buildbox-ng` - the buildbox used to build Teleport. It copies out the
third-party components from the previous image and installs whatever
other tools are needed by the build, whether from the distro archive
or directly from upstream (such as Go and Rust compilers).
Additionally, the three intermediate build stages of
`buildbox-thirdparty` can be built for working on the buildbox; `ctng`,
`compilers` and `tplibs`.
`buildbox-ng` will become just `buildbox` once it can replace the others
entirely, at which point those others will be removed.
Currently the build tools to build a FIPS version of teleport is not in
the buildbox. That has been troublesome as the rust boring crate wants
to build boringssl itself and it is not being done correctly with the
cross compilers. Further work is needed here, likely building boringssl
ourselves and pointing the rust boring crate at the pre-built libraries.
* buildbox: Add libelf.pc for pkg-config/pkconf
Add a libelf.pc file taken from elfutils and minorly adjusted for how we
install it separately. libelf was taken out of elfutils but did not take
the libelf.pc file, so we do that ourselves.
This will allow the build to use pkg-config/pkconf for selecting the
libraries to link teleport to as some later versions of libelf also need
`-lzstd`, such as this buildbox but also ubuntu-24.04.
* buildbox: Set prefix for zstd, add sh-cross-vars target
Set `PREFIX` when building `zstd` to ensure it's pkg-config file has the
correct prefix set in it, otherwise it has `-L/usr/local/lib` which we
do not want for cross-compiling.
Add a `sh-cross-vars` target to echo the cross-compiling vars for an
architecture, in a form that can be sourced by the shell:
eval $(make -s -f cross-compile.mk ARCH=arm64 sh-cross-vars)
This will set up your shell for cross-compiling for the given
architecture, which is useful when working on the libraries in the
buildbox.
* Address Roman's review comments
* Rename ctng to crosstoolng throughout
* Rename tplibs.mk to thirdparty-libs.mk
* Fix spelling mistakes.
* Address Jakub's review comments
* Remove `--hostname $(HOSTNAME)` when running containers.
* Remove `--volume /tmp:/tmp` when running containers.
* Rename bbcommon.mk to buildbox-common.mk
* buildbox: Fix up broken renaming
Renaming things was done poorly in the last few commits. Fix all that
up.
* buildbox: Use gold linker for arm64 builds
Use the `gold` linker when building arm64 binaries, as Enterprise
Teleport will not build with the binutils (bfd) linker; it gives
numerous errors when linking of the form:
something.rs: (.text.unlikely._XXX): relocation truncated to fit: R_AARCH64_CALL26 against symbol ...
These errors do not occur when using the `gold` linker, which is already
included and built by crosstool-NG.
* buildbox: Change crosstool arm tuple to match rust
Tweak the crosstool-NG configuration for the arm cross-compiler so that
the tuple for it matches the tuple currently used in the build by rust
for the same target. This is mostly to cause less confusion as there's
no real need for them to be different.
* buildbox: Add support for rust cross-compiling
Add environment variables so that cargo can find the appropriate
architecture-specific linker when cross-compiling. This is needed now as
we have a rust binary (fdpass-teleport) in the build, as opposed to just
rust compiled to a library linked to Go code. Rust does not have a cross
platform linker and relies an a linker in the toolchain for the target
architecture.
To make this work without needing per-architecture setup when building,
all of the toolchain binaries are symlinked into
/opt/thirdparty/host/bin so they are all in the path. Because each of
the binaries is prefixed with the target tuple, there are no name
clashes.
* buildbox: Add stages to buildbox Dockerfile for better caching
Download Go and Rust in separate stages and copy their installation into
the final container image. This helps as these stages no longer have
unrelated layers before them so they cache better. The Go and Rust
stages should only need to be rebuilt if the versions of the compilers
change, or some other aspect of the installation of the compilers
change. Previously adding a new package to be installed would cause Go
and Rust to be re-installed. This no longer happens.
* buildbox: Remove include of buildbox-common.mk in thirdparty-libs.mk
Remove the include of `buildbox-common.mk` from `thirdparty-libs.mk` as
it is included by `cross-compile.mk`. This caused some duplicate target
warnings when make ran.
* build: Add support for buildbox-ng to Makefile
Add supoprt for building in buildbox-ng to the build by including the
cross-compiling definitions if we are running in that buildbox and
gating some older cross-compiling definitions from being used.
* build: Add release targets using buildbox-ng
Add a set of targets to build releases using buildbox-ng instead of the
standard buildboxes. Ultimately these targets will be removed when the
old buildboxes are removed. These new targets are temporary until that
happens supporting testing of the new buildbox.
* buildbox: Add teleport user, expand comments
Add the teleport user (1000:1000) to the buildbox for running in CI
where a repository is checked-out inside the container instead of
mounting a volume on to /home/teleport. Make that home directory
world-writable so that it can still be used if 1000:1000 cannot be used
for some reason.
Expand and clean up comments.
Re-order ENV vars a little for better grouping.
Put Rust and Go temp directories under /tmp/build so a single volume can
be mounted there for persistent caches across runs.
* build: Fix invocation of pkg-config in $(shell ...)
Explicitly set `PKG_CONFIG_PATH` when running `$(PKGCONF)` as when
running that in a `$(shell ...)` expression, variables exported in the
Makefile are not exported for that shell, so `$(PKGCONF)` does not see
`PKG_CONFIG_PATH`. GNU make 4.4 changes this so that exported variables
are exported to `$(shell ...)` expressions, but the buildbox has GNU
make 4.3.
This was causing libbpf to not be detected properly so was building
teleport without bpf.- Loading branch information
