Fix Dockerfile to work on Silicon chip

[jpfleischer]

Before this change, doing docker build -t e-mission-server . would cause rosetta error: failed to open elf at /lib64/ld-linux-x86_64.so

https://docs.orbstack.dev/machines/#x86-programs

Therefore, we change the setup_config.sh to find the architecture of the host using uname -m and then download the right one from anaconda.com

Commands that were run:

$ docker build -t e-mission-server .

Console output

[+] Building 158.8s (20/20) FINISHED                                                                                            docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                            0.0s
 => => transferring dockerfile: 1.06kB                                                                                                          0.0s
 => WARN: MaintainerDeprecated: Maintainer instruction is deprecated in favor of using label (line 4)                                           0.0s
 => [internal] load metadata for docker.io/library/ubuntu:jammy-20240227                                                                        3.0s
 => [internal] load .dockerignore                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                 0.0s
 => [ 2/14] ADD https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem /etc/ssl/certs/                                               1.5s
 => [internal] load build context                                                                                                               1.4s
 => => transferring context: 272.49MB                                                                                                           1.4s
 => [ 1/14] FROM docker.io/library/ubuntu:jammy-20240227@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44eff17e                0.0s
 => CACHED [ 2/14] ADD https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem /etc/ssl/certs/                                        0.0s
 => CACHED [ 3/14] WORKDIR /usr/src/app                                                                                                         0.0s
 => CACHED [ 4/14] RUN apt-get -y -qq update                                                                                                    0.0s
 => CACHED [ 5/14] RUN apt-get install -y -qq curl                                                                                              0.0s
 => CACHED [ 6/14] RUN apt-get install -y -qq wget                                                                                              0.0s
 => CACHED [ 7/14] RUN apt-get install -y -qq git                                                                                               0.0s
 => CACHED [ 8/14] RUN apt-get install -y jq                                                                                                    0.0s
 => CACHED [ 9/14] RUN apt-get -y remove --purge build-essential                                                                                0.0s
 => CACHED [10/14] RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*                                                           0.0s
 => [11/14] COPY . .                                                                                                                            0.6s
 => [12/14] RUN chmod u+x ./.docker/setup_config.sh                                                                                             0.2s
 => [13/14] RUN bash -c "./.docker/setup_config.sh"                                                                                           148.9s
 => [14/14] RUN chmod u+x ./.docker/docker_start_script.sh                                                                                      0.1s 
 => exporting to image                                                                                                                          3.2s 
 => => exporting layers                                                                                                                         3.2s 
 => => writing image sha256:d6f7efb15f88dd4381edb9296c769ea13d1197f0d610284a63e52850497521da                                                    0.0s 
 => => naming to docker.io/library/e-mission-server                                                                                             0.0s 
                                                                                                                                                     
 1 warning found (use docker --debug to expand):
 - MaintainerDeprecated: Maintainer instruction is deprecated in favor of using label (line 4)

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview 

[shankari]

I am very skeptical that this is actually broken:

• GitHub actions uses an arm chip for macos-latest https://github.com/actions/runner-images/tree/main

• We use macos-latest for our CI/CD build
  https://github.com/e-mission/e-mission-server/blob/master/.github/workflows/osx-ubuntu-manual-install.yml

• The CI/CD build is passing
  https://github.com/e-mission/e-mission-server/actions/workflows/osx-ubuntu-manual-install.yml

Before I review this further, I would like some additional detailed evidence that this is actually broken and not just an error in your installation.

[shankari]

I am also not sure why you are building this container instead of using one of the pre-built images. We build an image on every commit to master.
https://github.com/e-mission/e-mission-server/actions/workflows/image_build_push.yml

[jpfleischer]

Please find my failing output attached.

me@mylaptop:~/e-mission-server$ uname -m
arm64
me@mylaptop:~/e-mission-server$ docker build -t e-mission-server .
[+] Building 29.7s (18/19)                                                                                                      docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                            0.0s
 => => transferring dockerfile: 1.06kB                                                                                                          0.0s
 => WARN: MaintainerDeprecated: Maintainer instruction is deprecated in favor of using label (line 4)                                           0.0s
 => [internal] load metadata for docker.io/library/ubuntu:jammy-20240227                                                                        3.0s
 => [internal] load .dockerignore                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                 0.0s
 => [ 2/14] ADD https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem /etc/ssl/certs/                                               1.7s
 => [ 1/14] FROM docker.io/library/ubuntu:jammy-20240227@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44eff17e                0.0s
 => [internal] load build context                                                                                                               0.1s
 => => transferring context: 193.78kB                                                                                                           0.1s
 => CACHED [ 2/14] ADD https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem /etc/ssl/certs/                                        0.0s
 => CACHED [ 3/14] WORKDIR /usr/src/app                                                                                                         0.0s
 => CACHED [ 4/14] RUN apt-get -y -qq update                                                                                                    0.0s
 => CACHED [ 5/14] RUN apt-get install -y -qq curl                                                                                              0.0s
 => CACHED [ 6/14] RUN apt-get install -y -qq wget                                                                                              0.0s
 => CACHED [ 7/14] RUN apt-get install -y -qq git                                                                                               0.0s
 => CACHED [ 8/14] RUN apt-get install -y jq                                                                                                    0.0s
 => CACHED [ 9/14] RUN apt-get -y remove --purge build-essential                                                                                0.0s
 => CACHED [10/14] RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*                                                           0.0s
 => [11/14] COPY . .                                                                                                                            0.8s
 => [12/14] RUN chmod u+x ./.docker/setup_config.sh                                                                                             0.2s
 => ERROR [13/14] RUN bash -c "./.docker/setup_config.sh"                                                                                      24.0s
------                                                                                                                                               
 > [13/14] RUN bash -c "./.docker/setup_config.sh":                                                                                                  
0.127 About to start conda update, this may take some time...                                                                                        
0.127 Installing for version 23.5.2 and platform Linux-x86_64                                                                                        
0.156   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                
0.156                                  Dload  Upload   Total   Spent    Left  Speed                                                                  
100 89.0M  100 89.0M    0     0  3881k      0  0:00:23  0:00:23 --:--:-- 4587k
23.69 PREFIX=/root/miniconda-23.5.2
23.92 Unpacking payload ...
23.92 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
23.92  miniconda.sh: line 353:    40 Exit 141                extract_range $boundary1 $boundary2
23.92         41 Trace/breakpoint trap   | "$CONDA_EXEC" constructor --extract-tarball --prefix "$PREFIX"
------

 1 warning found (use docker --debug to expand):
 - MaintainerDeprecated: Maintainer instruction is deprecated in favor of using label (line 4)
Dockerfile:30
--------------------
  28 |     
  29 |     # # This clone puts the server code into the image, not the container
  30 | >>> RUN bash -c "./.docker/setup_config.sh"
  31 |     
  32 |     # #declare environment variables
--------------------
ERROR: failed to solve: process "/bin/sh -c bash -c \"./.docker/setup_config.sh\"" did not complete successfully: exit code: 133
me@mylaptop:~/e-mission-server$ 

[shankari]

This doesn't answer my question - why are you trying to build docker in the first place instead of using a pre-built image?

[jpfleischer]

The prebuilt images, as found at:
https://hub.docker.com/r/shankari/e-mission-server/tags
Do indeed work on my M2. However it is because of emulation.

My console output

$ docker pull shankari/e-mission-server:master_2024-10-18--38-04
master_2024-10-18--38-04: Pulling from shankari/e-mission-server
bccd10f490ab: Already exists 
76b379fcb29c: Pull complete 
e24df673655c: Pull complete 
d3105dae1e6a: Pull complete 
27f386dc5bfa: Pull complete 
7817ed6eed9d: Pull complete 
4087507cb8fc: Pull complete 
fef30a078770: Pull complete 
4f4fb700ef54: Pull complete 
c5c914168641: Pull complete 
72f8632b278d: Pull complete 
d04958ebf792: Pull complete 
b149a77d98fd: Pull complete 
676f684a3513: Pull complete 
Digest: sha256:e5c6396382735b0925b4fc26a2be839253febd8796efcf900ea002225ee598d6
Status: Downloaded newer image for shankari/e-mission-server:master_2024-10-18--38-04
docker.io/shankari/e-mission-server:master_2024-10-18--38-04

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview shankari/e-mission-server:master_2024-10-18--38-04

This downloads ok, but upon running:

$ docker run -d --name emission-server -p 8080:8080 shankari/e-mission-server:master_2024-10-18--38-04
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
6537b9fb608c264def49797d7ad2ac1a446020ce6f1fbbb75ac328aeb17d1669
$

..which is still OK. It runs! But it seems trivial to support an arm version too.

While it is true that the runner macos-latest uses ARM, and even this workflow action shows no error, https://github.com/e-mission/e-mission-server/actions/runs/11409936953/job/31751179855
it is not testing for Ubuntu on ARM. When I try to build the docker image, and the Docker image uses ubuntu, and my laptop is ARM, it fails.

The issue gets weirder, since I told a workflow to run on macOS, get Docker, and try building the image. (It is virtual machines all the way down!)
This is while keeping the preexisting, unchanged version of setup_config.sh and the runner, like my laptop, complained of x86 instructions: qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
https://github.com/e-mission/e-mission-server/actions/runs/11414440492/job/31763344412

Well we actually know why. It is because it was making a docker container, that used Ubuntu, on arm.

Platform	x86_64	arm64
macOS	✅	✅
Ubuntu	✅	❌

For whatever reason Mac x86 conda works on both architectures, but linux x86 does not.
Is it Rosetta emulation on Mac ?

My suggestion

diff --git a/.docker/setup_config.sh b/.docker/setup_config.sh
index 5a1e0d42..e8d04e8c 100644
--- a/.docker/setup_config.sh
+++ b/.docker/setup_config.sh
@@ -1,6 +1,15 @@
 set -e
 echo "About to start conda update, this may take some time..."
-source setup/setup_conda.sh Linux-x86_64
+ARCH=$(uname -m)
+if [ "$ARCH" == "x86_64" ]; then
+    source setup/setup_conda.sh Linux-x86_64
+elif [ "$ARCH" == "aarch64" ]; then
+    source setup/setup_conda.sh Linux-aarch64
+else
+    echo "Unsupported architecture: $ARCH"
+    exit 1
+fi
+
 # now install the emission environment
 source setup/setup.sh

Even consider my more elaborate suggestion in e-mission-common
https://github.com/JGreenlee/e-mission-common/pull/9/files#diff-c44c478ef8ea22b3a6e6a9d7fe675efc54a06b36d95d0bac6a74c755c15fdfb2

No emulation would make things faster maybe. Maybe not too much to notice a difference 😛

[jpfleischer]

We don't intend to fix this