Project infrastructure and first Dockerfile

[mythmon]

This is a prototype of the default environment that cloud and on-prem builders will use to build Framework projects. This doesn't automatically build or publish anywhere.

The goal is "everything in the docs". This PR doesn't get us there. It has

• Node 20
• Python 3.11
• R 4.3
• duckdb 0.10.1
• Rust 1.77

We probably also want to add (based on our docs)

• Go
• Java
• Julia
• PHP

This includes tests that the expected binaries are available in the image. They can take a while to run at first, since they they first build the docker image.

To use this image this yourself, you can run yarn build, and then docker run -it observablehq/framework-runtime:latest to get a shell in the runtime.

[mythmon]

One thing to note is that a lot of the code was generated by the Yarn Berry init process. Please focus on Dockerfile, test.ts, and package.json.

[mythmon]

This is a public key used to validate the Node.js installation in the Dockerfile

[mythmon]

This is a public key used to validate the R installation in the Dockerfile

[Fil]

I did docker build . and saw:

~/Source/framework-runtime mythmon/240329/initial*
❯ docker build .
[+] Building 42.0s (17/23)
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 3.08kB                                     0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/buildpack-deps:bookwor  2.0s
 => [internal] preparing inline document                                   0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 4.90kB                                        0.0s
 => [base 1/2] FROM docker.io/library/buildpack-deps:bookworm@sha256:33f  34.6s
 => => resolve docker.io/library/buildpack-deps:bookworm@sha256:33fd523e6  0.0s
 => => sha256:33fd523e6d918b2d066038992f814ff881eb6de323d 1.86kB / 1.86kB  0.0s
 => => sha256:236f29b4631548c327b2d1d0590ef8011bff534a87f 1.17kB / 1.17kB  0.0s
 => => sha256:02567629c9e9c4dc92672656620fd7750ba05eb4c4f 3.72kB / 3.72kB  0.0s
 => => sha256:6ee0baa58a3d368515336c1b5c1cade29c975e1b 49.59MB / 49.59MB  10.6s
 => => sha256:992a857ef57584af4efb4c62d68456f1e8513c95d 23.58MB / 23.58MB  5.1s
 => => sha256:3861a6536e4e911503e7d2fc8f93228491ba45d1 63.99MB / 63.99MB  21.2s
 => => sha256:e5e6faea05ead1ac9cd3244827816e2385b0d6 202.54MB / 202.54MB  29.8s
 => => extracting sha256:6ee0baa58a3d368515336c1b5c1cade29c975e1b49a832f1  1.3s
 => => extracting sha256:992a857ef57584af4efb4c62d68456f1e8513c95d6248fd7  0.3s
 => => extracting sha256:3861a6536e4e911503e7d2fc8f93228491ba45d1e5def0d2  1.6s
 => => extracting sha256:e5e6faea05ead1ac9cd3244827816e2385b0d62299f7937a  4.5s
 => [internal] preparing inline document                                   0.0s
 => [base 2/2] RUN apt update                                              2.4s
 => CANCELED [python 1/1] RUN --mount=type=cache,target=/var/cache/apt,id  2.9s
 => CANCELED [duckdb 1/1] RUN cd $(mktemp -d)     && wget https://github.  2.9s
 => CANCELED [rust 1/2] RUN set -eux;     dpkgArch="$(dpkg --print-archit  2.9s
 => [r 1/3] COPY r-project.gpg /etc/apt/keyrings/r-project.gpg             0.0s
 => [node 1/3] COPY nodesource.gpg /etc/apt/keyrings/nodesource.gpg        0.0s
 => [r 2/3] COPY <<EOF /etc/apt/sources.list.d/r-project.list              0.0s
 => [node 2/3] COPY <<EOF /etc/apt/sources.list.d/nodesource.list          0.0s
 => ERROR [r 3/3] RUN --mount=type=cache,target=/var/cache/apt,id=framewo  2.8s
 => CANCELED [node 3/3] RUN --mount=type=cache,target=/var/cache/apt,id=f  2.8s
------
 > [r 3/3] RUN --mount=type=cache,target=/var/cache/apt,id=framework-runtime-r     apt update     && apt install -y --no-install-recommends       r-base-core       r-recommended:
#0 0.349
#0 0.349 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
#0 0.349
#0 0.412 Hit:1 http://deb.debian.org/debian bookworm InRelease
#0 0.412 Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
#0 0.425 Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
#0 0.808 Get:4 https://cloud.r-project.org/bin/linux/debian bookworm-cran40/ InRelease [4367 B]
#0 1.079 Get:5 https://cloud.r-project.org/bin/linux/debian bookworm-cran40/ Sources [20.0 kB]
#0 1.441 Get:6 https://cloud.r-project.org/bin/linux/debian bookworm-cran40/ Packages [24.6 kB]
#0 1.456 Fetched 48.9 kB in 1s (45.2 kB/s)
#0 1.456 Reading package lists...
#0 1.811 Building dependency tree...
#0 1.956 Reading state information...
#0 1.980
#0 1.980 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
#0 1.980
#0 1.980 11 packages can be upgraded. Run 'apt list --upgradable' to see them.
#0 1.980 Reading package lists...
#0 2.407 Building dependency tree...
#0 2.498 Reading state information...
#0 2.538 Some packages could not be installed. This may mean that you have
#0 2.538 requested an impossible situation or if you are using the unstable
#0 2.538 distribution that some required packages have not yet been created
#0 2.538 or been moved out of Incoming.
#0 2.538 The following information may help to resolve the situation:
#0 2.538
#0 2.538 The following packages have unmet dependencies:
#0 2.593  r-recommended : Depends: r-base-core (>= 4.3.3-2~bookwormcran.0) but 4.2.2.20221110-2 is to be installed
#0 2.593                  Depends: r-cran-boot (>= 1.2.19) but it is not going to be installed
#0 2.600 E: Unable to correct problems, you have held broken packages.
------
Dockerfile:40
--------------------
  39 |     EOF
  40 | >>> RUN --mount=type=cache,target=/var/cache/apt,id=framework-runtime-r \
  41 | >>>     apt update \
  42 | >>>     && apt install -y --no-install-recommends \
  43 | >>>       r-base-core \
  44 | >>>       r-recommended
  45 |
--------------------
ERROR: failed to solve: process "/bin/sh -c apt update     && apt install -y --no-install-recommends       r-base-core       r-recommended" did not complete successfully: exit code: 100

it was possible to build a subset with:

❯ docker build . --target node -t test

❯ docker run -it test bash
root@53e99e7ad046:/# node -v
v20.12.1

[Fil]

add typescript?

[mythmon]

Typescript is basically a Node.js library, which can be installed like any other JS library. What would you expect the image to do to support that better?

[Fil]

Sure. I thought mentioning it would be useful.

[Fil]

does this work on macos? I have a M1 that says arm64, and if I build it it won't run duckdb

❯ docker build . --target duckdb -t test-duckdb

❯ docker run -it test-duckdb bash
root@52515696792a:/# duckdb -v
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

[mythmon]

Fabian was also having trouble with arm64. Our primary targets for this image are all Linux-x64, so I'm not concerned about fixing this immediately (though I understand it makes it harder to review).

I think we should probably merge this PR and then have someone with an arm64 system work on adding support for that in another PR.

[Fil]

makes sense; fwiw I read:

❯ uname -p
arm
❯ uname -o
Darwin

[mythmon]

I added support to pull the arm64 version of duckdb, the same way that Rust does. That might help here.

[Fil]

It would be nice to have a small tutorial (or read-through) explaining how to build and use this image (a friend helped me).

Also I clicked around and saw some very frightening red badges.

[mythmon]

@Fil what tool is that, and what do those red and yellow shields mean? Your Docker tools are very different from mine, so I might require some hand holding about what you're seeing.

[mootari]

That looks like the vulnerabilities view in Docker Desktop's dashboard:

[Fil]

@Fil what tool is that

I absolutely don't know, I don't remember installing anything particular. I need to find time to investigate.

EDIT: just saw Fabian's screenshot; that's it.

[mootari]

I can build and run the image with

yarn build --platform linux/amd64
docker run --platform linux/amd64 -it observablehq/framework-runtime:latest

[Fil]

Still no luck with R for me (same error as before) when I do yarn build on macos/M1

#0 2.807  r-recommended : Depends: r-base-core (>= 4.3.3-2~bookwormcran.0) but 4.2.2.20221110-2 is to be installed
#0 2.807                  Depends: r-cran-boot (>= 1.2.19) but it is not going to be installed
#0 2.814 E: Unable to correct problems, you have held broken packages.

with yarn build --platform linux/amd64 (like Fabian suggests) I get a working build, but note that the rust-script compilation step (RUN cargo install rust-script) takes a looong time (200+s — maybe it doesn't matter, though). The image itself seems to run slowly, maybe because of the non-native platform (?). But again, since the image is intended to run on PCs it should run fast in the cloud.

Then I can do:

# cd home
# npm init @observablehq
...

I'm not sure if and how I can expose the :3000 port to see the preview from my browser. But I could successfully run npm run build, Rscript, python3, duckdb v0.10.1, so I guess everything's working as intended.

[mythmon]

I think the problem with R is a bigger one than I can fix here. The problem is that the R project doesn't seem to have published packages for 4.3 for arm64 yet. I think we can accept that his doesn't work on arm64 right now, or we can downgrade to R 4.2.

but note that the rust-script compilation step (RUN cargo install rust-script) takes a looong time (200+s — maybe it doesn't matter, though)

This isn't surprising. The installation process for rust-script compile from source, and like you mention, running in non-native mode is going to be slower. Actually any Rust-in-Docker is going to be slower than native, because of the extra translation layers of the Docker VM. Not much we can do about that though. This isn't a time our users will see, though it will affect our CI.

I'm not sure if and how I can expose the :3000 port to see the preview from my browser.

When running the container you can add -p 3000:3000 to publish the port from the container to your system, and then use localhost:3000 as normal. However, I think you'll need to get Framework to listen on a different hostname. By default it only listens on 127.0.0.1, which wasn't accessible from outside the container for me. I don't know how to configure Framework to do that, so I left it out of these instructions.

[Fil]

By default it only listens on 127.0.0.1, which wasn't accessible from outside the container for me. I don't know how to configure Framework to do that, so I left it out of these instructions.

It is the host option. I've tried to run the following:

# npm init @observablehq
# cd hello-framework/
# yarn dev --host 0.0.0.0

but from the host machine (the Mac) I can't connect to the dev server (on the container—I supposed it should work with curl 172.17.0.2:3000, at least that's the IP address I see in the container's /etc/hosts).

I find it hard to debug with so few tools available (no ping, no ifconfig). Are we being too strict with the requirements? "everything in the docs" does not mean "nothing outside of it" :)

[mythmon]

The 172... address is local to the Docker VM, and won't be accessible to your host system. If you are using -p to expose the port, then you should be able to access it in your browser at localhost:3000.

[mythmon]

In the interest of keeping this PR from getting even bigger, I'd like to merge this without:

• arm64 support
• figuring out if yarn dev works
• any additional CLI tools

and instead save all that for follow up PRs.

@Fil @mootari do you have any objections to that? Are there any more things you'd like to do here before we merge?

[Fil]

Here's a complete playbook on macOS/M2:

host> yarn build --platform linux/amd64
host> docker run -p 3010:3010 --platform linux/amd64 -it observablehq/framework-runtime:latest
container# cd home
container# npm init @observablehq
 … / …
container# cd hello-observable
container# npm run dev -- --host=0.0.0.0 --port=3010

then open http://localhost:3010/

Note that I tried several commands before I could find this one. The suggest help command ( See 'observable help preview') doesn't work (this looks like a general problem, not linked to this docker file PR.)

[Fil]

I just noticed that this image has perl, which seems useful, but had not been explicitly listed in the requirements yet.

[mythmon]

Congrats on getting it working! Hopefully we can make that easier in the future.

I think perl is a required part of Debian.

[Fil]

My point about perl is that, since we have it, we should document it (add it to the list of elements that users will know are here).