Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Host assembly file used when cross-compiling for aarch64 #761

Closed
zeenix opened this issue Nov 4, 2024 · 7 comments
Closed

Host assembly file used when cross-compiling for aarch64 #761

zeenix opened this issue Nov 4, 2024 · 7 comments
Labels
bug This issue is a bug. dependencies This issue is a problem in a dependency. p3 This is a minor priority issue

Comments

@zeenix
Copy link

zeenix commented Nov 4, 2024

Describe the bug

While trying to cross-compile for aarch64 on an x86-64 Linux machine, it seems that wrong assembly files get copied over and compilation fails with a huge amount of errors:

Copying platform assembly files from /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-lc/generated-src/linux-x86_64/crypto/ to /target/aarch64-unknown-linux-gnu/debug/build/greengrass-sdk-rs-1bdae08b7f817854/out/build/deps/AWSLC/crypto
  /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S: Assembler messages:
  /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:302: Error: unknown pseudo-op: `.att_syntax'
  /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-
...

The cmake command is:

cmake "/home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2" \
    "-DBUILD_SHARED_LIBS=OFF" \
    "-DCMAKE_CROSSCOMPILING=TRUE" \
    "-DCMAKE_SYSTEM_NAME=Linux" \
    "-DCMAKE_SYSTEM_PROCESSOR=aarch64" \
    "-DCMAKE_INSTALL_PREFIX=/target/aarch64-unknown-linux-gnu/debug/build/greengrass-sdk-rs-1bdae08b7f817854/out" \
    "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections" \
    "-DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc" \
    "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections" \
    "-DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++" \
    "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections" \
    "-DCMAKE_ASM_COMPILER=/usr/bin/aarch64-linux-gnu-gcc" \
    "-DCMAKE_BUILD_TYPE=Debug"

Is it possible there is some cmake variable that is needed to be set?

Expected Behavior

The right assembly files dir is copied over and build succeeds.

Current Behavior

Incorrect assembly files get copied and compilation fails.

Reproduction Steps

Run the cmake command in the description above.

Possible Solution

Document how to cross-build in the README or make the build work with the command specified in the description.

Additional Information/Context

No response

SDK version used

1.34 (git revision: 602121d)

Environment details (OS name and version, etc.)

Fedora 40, building under Cargo cross container env (uses Ubuntu images)
@zeenix zeenix added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 4, 2024
@jmklix
Copy link
Member

jmklix commented Nov 6, 2024

It looks like you are having a problem with one of the dependencies: aws-lc
aws-lc doesn't seem to be correctly cross compiling for arm.

Looking on through the aws-lc build docs, they mention an additional cmake parameter that you might want to pass:

If you want to cross-compile then there is an example toolchain file for 32-bit Intel in util/. Wipe out the build directory, run cmake like this:

cmake -B build -DCMAKE_TOOLCHAIN_FILE=../util/32-bit-toolchain.cmake -GNinja

I would try adding that to your cmake command and see if that works. Also seeing if you can cross compile aws-lc in your environment on it's own would be helpful.

@jmklix jmklix added dependencies This issue is a problem in a dependency. p3 This is a minor priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 2 days. and removed needs-triage This issue or PR still needs to be triaged. labels Nov 6, 2024
@zeenix
Copy link
Author

zeenix commented Nov 6, 2024 •
edited
Loading

@jmklix Thanks! I just treat the whole repo as one self-contained project so I didn't even think to check for a README in aws-lc lib. :)

I would try adding that to your cmake command and see if that works.

I was about to try it but looking at the content of that .cmake file being passed, it is only setting cmake variables, just like I am in my command. So that can't work (especially since that's setting the arch to be 32-bit x86, while I'm targetting 64-bit ARM).

Also seeing if you can cross compile aws-lc in your environment on it's own would be helpful.

I'm trying that right now. It's not trivial though since for some reason, it starts to require Go compiler if I build it standalone.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 2 days. label Nov 6, 2024
@zeenix
Copy link
Author

zeenix commented Nov 6, 2024

Also seeing if you can cross compile aws-lc in your environment on it's own would be helpful.

I'm trying that right now. It's not trivial though since for some reason, it starts to require Go compiler if I build it standalone

The Go compiler was too old for this to work. I instead went for building against the system openssl and that seems to work. Thanks again for your time.

@zeenix zeenix closed this as completed Nov 6, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 6, 2024

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

@jmklix
Copy link
Member

jmklix commented Nov 6, 2024

Can you share how you got it working with openssl? In case someone in the future runs across this issue and is trying to do the same thing

@zeenix
Copy link
Author

zeenix commented Nov 6, 2024

Can you share how you got it working with openssl? In case someone in the future runs across this issue and is trying to do the same thing

Actually, it turns out that I only got it to build (by setting "-DUSE_OPENSSL=ON") but later when I try to link my library, I get a lot of undefined symbols errors:

error: linking with `aarch64-linux-gnu-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "aarch64-linux-gnu-gcc" "/tmp/rustc0odEaN/symbols.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.06qqxrplnqir8hco1cf4f80s6.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.0k1zl0jx60gl2hu2h4qr41m9g.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.0zoxo5lf83lbbuaxh7dnx37r6.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.1fj6eb8560p9cmuynivmgf566.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.1pnxvczk94axri79jwmmwjg57.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.23tfjerv1y26ne1x1s2awqpgg.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.286kxbgyggtpf7it3fx53iyfb.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.2hlj3c57cx6r8vqzlrkxq50ui.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.43omxd8vowghmf8adx4734ham.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.4ex84xf0vm06myv8f1pl6ek7l.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.53rmi0rsn8i1ngzp149aq6ia0.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.78t51wx4lkuv04ifj75d1e3ke.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.8pfvvoa7k36ugp1ehjp1bndag.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.8tu9u0zhhd30embndqkbutcib.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.9nzh1xqn292ody694xaqo2dog.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.alxeytl1ot4wnb4pfyar7lruy.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.amebz30o6tvwutjov6p4lhdg5.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.bwp4w4fsoyzt6a7w0szgup56u.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.cgdsrk14hf5z2b3pbl45zqjn8.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.dkpt4mto1mqeetx6i550wpzut.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.eskhcwyjgcqre858o6q4r0cn6.rcgu.o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10.8bgb893rohvsfc9muk85l97gj.rcgu.o" "-Wl,--as-needed" "-L" "/target/aarch64-unknown-linux-gnu/debug/deps" "-L" "/target/debug/deps" "-L" "/target/aarch64-unknown-linux-gnu/debug/build/greengrass-sdk-rs-1bdae08b7f817854/out/lib64" "-L" "/target/aarch64-unknown-linux-gnu/debug/build/greengrass-sdk-rs-1bdae08b7f817854/out/lib" "-L" "/target/aarch64-unknown-linux-gnu/debug/build/greengrass-sdk-rs-1bdae08b7f817854/out" "-L" "/target/aarch64-unknown-linux-gnu/debug/build/cxx-95542bd5f3eac3a1/out" "-L" "/target/aarch64-unknown-linux-gnu/debug/build/link-cplusplus-e561643d18d9d1aa/out" "-L" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bdynamic" "-lcrypto" "-lssl" "-Wl,-Bstatic" "-Wl,--whole-archive" "-ls2n" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-io" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-iot" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-common" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-event-stream" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-checksums" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-cal" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-s3" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-mqtt" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-auth" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-http" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-compression" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-c-sdkutils" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-laws-crt-cpp" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-lGreengrassIpc-cpp" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-lEventstreamRpc-cpp" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-lgreenrass-ipc" "-Wl,--no-whole-archive" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libtest-1955698392f0439a.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgetopts-60e558559cea038a.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunicode_width-10ecd85e0abc6461.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_std-2fe35ba938020354.rlib" "/target/aarch64-unknown-linux-gnu/debug/deps/libcxx-1124978dd72829b0.rlib" "/target/aarch64-unknown-linux-gnu/debug/deps/liblink_cplusplus-0634ac7948ec81c5.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-2bf0b2a5e0a60917.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-0af01d78b15f6872.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-aa90d1efd19541cb.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-6645a3a6124c47a1.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-3de13e717f4d9e74.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-f50e3ac5e8bc32ca.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-f84a4f82a7a57e94.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-bd992eebc2a12fc4.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-c9882005b74b1193.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-b18e8234ebc582c8.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-79ef105ee0e8243e.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-652182712f7d3bc4.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-6cb747324af00512.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-740a433abf104d06.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-1e2f311c277b60cf.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-85299feea58ac1e7.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-2a73a86214747017.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-29cdff63f523de0d.rlib" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-405c9891256dbf91.rlib" "-Wl,-Bdynamic" "-lstdc++" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/zeenix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/target/aarch64-unknown-linux-gnu/debug/deps/greengrass_sdk_rs-e441c1ed2177fe10" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
: in function `s2n_aead_cipher_aes128_gcm_available':
          /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/s2n/crypto/s2n_aead_cipher_aes_gcm.c:30: undefined reference to `EVP_aes_128_gcm'
          /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /target/aarch64-unknown-linux-gnu/debug/build/greengrass-sdk-rs-1bdae08b7f817854/out/lib/libs2n.a(s2n_aead_cipher_aes_gcm.c.o): in function `s2n_aead_cipher_aes256_gcm_available':
          /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/s2n/crypto/s2n_aead_cipher_aes_gcm.c:39: undefined reference to `EVP_aes_256_gcm'
          /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /target/aarch64-unknown-linux-gnu/debug/build/greengrass-sdk-rs-1bdae08b7f817854/out/lib/libs2n.a(s2n_aead_cipher_aes_gcm.c.o): in function `s2n_aead_cipher_aes_gcm_encrypt':
          /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/s2n/crypto/s2n_aead_cipher_aes_gcm.c:216: undefined reference to `EVP_EncryptInit_ex'
          /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/s2n/crypto/s2n_aead_cipher_aes_gcm.c:225: undefined reference to `EVP_EncryptUpdate'
          /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/s2n/crypto/s2n_aead_cipher_aes_gcm.c:228: undefined reference to `EVP_EncryptUpdate'
          /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /home/zeenix/checkout/jucr-io/greengrass-sdk-rs/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/s2n/crypto/s2n_aead_cipher_aes_gcm.c:234: undefined reference to `EVP_EncryptFinal_ex'
...

even though libssl and libcrypto are linked.

@zeenix
Copy link
Author

zeenix commented Nov 6, 2024

Actually, it turns out that I only got it to build (by setting "-DUSE_OPENSSL=ON") but later when I try to link my library, I get a lot of undefined symbols errors:

Seems like each time, I comment here things change immediately after. :) If I link my library against another (Rust) binary, it seems to work just fine. 🤷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. dependencies This issue is a problem in a dependency. p3 This is a minor priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zeenix @jmklix