Refactor `meta.jq` to prepare for signing and `oci-import` implementation #17

tianon · 2024-01-25T21:31:39Z

explicit "containerd image storage in dockerd" placeholder
explicit "annotations" helper/generator (so they can be added in the oci-import builder later)
setting the org.opencontainers.image.created annoation inside the image index to the actual build date (otherwise SOURCE_DATE_EPOCH makes it hard to tell an image is actually freshly built)
more whitespace in generated commands for better readability
extract OCI tar in the build step (so the "output" of the build is an OCI layout directory vs tarball) instead of the pull step (so we can more ergonomically add pre-push signing)
use buildx's --annotation flag for making annotations easier to read

tianon · 2024-01-25T22:47:32Z

An example generated build command:

# jq -L .scripts 'include "meta"; first(.[]) | build_command' -r ../builds.json
SOURCE_DATE_EPOCH=1705382289 \
	docker buildx build --progress=plain \
	--provenance=mode=max \
	--output '"type=oci","dest=temp.tar"' \
	--annotation 'org.opencontainers.image.source=https://github.com/tianon/docker-bash.git#b08b986ade4dab726334b111c89688491534b732:devel' \
	--annotation 'org.opencontainers.image.revision=b08b986ade4dab726334b111c89688491534b732' \
	--annotation 'org.opencontainers.image.created=2024-01-16T05:18:09Z' \
	--annotation 'org.opencontainers.image.version=devel-20240114' \
	--annotation 'org.opencontainers.image.url=https://hub.docker.com/_/bash' \
	--annotation 'manifest-descriptor:org.opencontainers.image.source=https://github.com/tianon/docker-bash.git#b08b986ade4dab726334b111c89688491534b732:devel' \
	--annotation 'manifest-descriptor:org.opencontainers.image.revision=b08b986ade4dab726334b111c89688491534b732' \
	--annotation 'manifest-descriptor:org.opencontainers.image.created=2024-01-25T22:46:52Z' \
	--annotation 'manifest-descriptor:org.opencontainers.image.version=devel-20240114' \
	--annotation 'manifest-descriptor:org.opencontainers.image.url=https://hub.docker.com/_/bash' \
	--tag 'bash:devel-20240114' \
	--tag 'bash:devel' \
	--tag 'bash:devel-20240114-alpine3.19' \
	--tag 'bash:devel-alpine3.19' \
	--tag 'amd64/bash:devel-20240114' \
	--tag 'amd64/bash:devel' \
	--tag 'amd64/bash:devel-20240114-alpine3.19' \
	--tag 'amd64/bash:devel-alpine3.19' \
	--tag 'oisupport/staging-amd64:dbfd320b24fdbcf3e17174640287049bde81906f59592b6d4e3e42f47f9b5517' \
	--platform 'linux/amd64' \
	--build-context 'alpine:3.19=docker-image://alpine:3.19@sha256:13b7e62e8df80264dbb747995705a986aa530415763a6c58f84a3ca8af9a5bcd' \
	--build-arg BUILDKIT_SYNTAX="$BASHBREW_BUILDKIT_SYNTAX" \
	--file 'Dockerfile' \
	'https://github.com/tianon/docker-bash.git#b08b986ade4dab726334b111c89688491534b732:devel'
mkdir temp
tar -xvf temp.tar -C temp
rm temp.tar
jq '
	.manifests |= (
		del(.[].annotations)
		| unique
		| if length != 1 then
			error("unexpected number of manifests: " + length)
		else . end
	)
' temp/index.json > temp/index.json.new
mv temp/index.json.new temp/index.json

tianon · 2024-01-25T23:12:27Z

Realizing we could (ab)use our sources.json and builds.json test suite to generate some example commands in the repo itself so that changes like this actually show what the pull/build/push command change is (and helps illustrate why the changes are being made). 👀

So, feel free to review this, but I think I'm going to work on that as a separate thing so I can use this PR as the guinea pig for showing off the result of it.

tianon · 2024-01-25T23:41:28Z

Rebased on top of #18 so the diff to .test/example-commands.sh is more interesting/useful. 👀

…tion - explicit "containerd image storage in dockerd" placeholder - explicit "annotations" helper/generator (so they can be added in the `oci-import` builder later) - setting the `org.opencontainers.image.created` annoation inside the image index to the actual build date (otherwise `SOURCE_DATE_EPOCH` makes it hard to tell an image is actually freshly built) - more whitespace in generated commands for better readability - extract OCI tar in the build step (so the "output" of the build is an OCI layout directory vs tarball) instead of the pull step (so we can more ergonomically add pre-push signing) - use buildx's `--annotation` flag for making annotations easier to read

tianon · 2024-01-25T23:50:52Z

.test/example-commands.sh

@@ -3,15 +3,49 @@

 # </pull>
 # <build>
-SOURCE_DATE_EPOCH=1700741054 docker buildx build --progress=plain --provenance=mode=max --sbom=generator="$BASHBREW_BUILDKIT_SBOM_GENERATOR" --output '"type=oci","dest=temp.tar","annotation.org.opencontainers.image.source=https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli","annotation-manifest-descriptor.org.opencontainers.image.source=https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli","annotation.org.opencontainers.image.revision=6d541d27b5dd12639e5a33a675ebca04d3837d74","annotation-manifest-descriptor.org.opencontainers.image.revision=6d541d27b5dd12639e5a33a675ebca04d3837d74","annotation.org.opencontainers.image.version=24.0.7-cli","annotation-manifest-descriptor.org.opencontainers.image.version=24.0.7-cli","annotation.org.opencontainers.image.url=https://hub.docker.com/_/docker","annotation-manifest-descriptor.org.opencontainers.image.url=https://hub.docker.com/_/docker"' --tag 'docker:24.0.7-cli' --tag 'docker:24.0-cli' --tag 'docker:24-cli' --tag 'docker:cli' --tag 'docker:24.0.7-cli-alpine3.18' --tag 'oisupport/staging-amd64:4b199ac326c74b3058a147e14f553af9e8e1659abc29bd3e82c9c9807b66ee43' --platform 'linux/amd64' --build-context 'alpine:3.18=docker-image://alpine:3.18@sha256:d695c3de6fcd8cfe3a6222b0358425d40adfd129a8a47c3416faff1a8aece389' --build-arg BUILDKIT_SYNTAX="$BASHBREW_BUILDKIT_SYNTAX" --file 'Dockerfile' 'https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli'


This change is much cuter with --ignore-all-space --word-diff:

Diff:

diff --git a/.test/example-commands.sh b/.test/example-commands.sh index f165244..4bcbaf8 100644 --- a/.test/example-commands.sh +++ b/.test/example-commands.sh @@ -3,15 +3,49 @@ # </pull> # <build> SOURCE_DATE_EPOCH=1700741054 {+\+} docker buildx build --progress=plain {+\+} --provenance=mode=max {+\+} --sbom=generator="$BASHBREW_BUILDKIT_SBOM_GENERATOR" {+\+} --output [-'"type=oci","dest=temp.tar","annotation.org.opencontainers.image.source=https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli","annotation-manifest-descriptor.org.opencontainers.image.source=https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli","annotation.org.opencontainers.image.revision=6d541d27b5dd12639e5a33a675ebca04d3837d74","annotation-manifest-descriptor.org.opencontainers.image.revision=6d541d27b5dd12639e5a33a675ebca04d3837d74","annotation.org.opencontainers.image.version=24.0.7-cli","annotation-manifest-descriptor.org.opencontainers.image.version=24.0.7-cli","annotation.org.opencontainers.image.url=https://hub.docker.com/_/docker","annotation-manifest-descriptor.org.opencontainers.image.url=https://hub.docker.com/_/docker"'-]{+'"type=oci","dest=temp.tar"' \+} {+ --annotation 'org.opencontainers.image.source=https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli' \+} {+ --annotation 'org.opencontainers.image.revision=6d541d27b5dd12639e5a33a675ebca04d3837d74' \+} {+ --annotation 'org.opencontainers.image.created=2023-11-23T12:04:14Z' \+} {+ --annotation 'org.opencontainers.image.version=24.0.7-cli' \+} {+ --annotation 'org.opencontainers.image.url=https://hub.docker.com/_/docker' \+} {+ --annotation 'manifest-descriptor:org.opencontainers.image.source=https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli' \+} {+ --annotation 'manifest-descriptor:org.opencontainers.image.revision=6d541d27b5dd12639e5a33a675ebca04d3837d74' \+} {+ --annotation 'manifest-descriptor:org.opencontainers.image.created=1970-01-01T00:00:00Z' \+} {+ --annotation 'manifest-descriptor:org.opencontainers.image.version=24.0.7-cli' \+} {+ --annotation 'manifest-descriptor:org.opencontainers.image.url=https://hub.docker.com/_/docker' \+} --tag 'docker:24.0.7-cli' {+\+} --tag 'docker:24.0-cli' {+\+} --tag 'docker:24-cli' {+\+} --tag 'docker:cli' {+\+} --tag 'docker:24.0.7-cli-alpine3.18' {+\+} --tag 'oisupport/staging-amd64:4b199ac326c74b3058a147e14f553af9e8e1659abc29bd3e82c9c9807b66ee43' {+\+} --platform 'linux/amd64' {+\+} --build-context 'alpine:3.18=docker-image://alpine:3.18@sha256:d695c3de6fcd8cfe3a6222b0358425d40adfd129a8a47c3416faff1a8aece389' {+\+} --build-arg BUILDKIT_SYNTAX="$BASHBREW_BUILDKIT_SYNTAX" {+\+} --file 'Dockerfile' {+\+} 'https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/cli'[-# </build>-] [-# <push>-] mkdir temp tar -xvf temp.tar -C temp {+rm temp.tar+} jq [-'.manifests-]{+'+} {+ .manifests+} |= [-(del(.[].annotations)-]{+(+} {+ del(.[].annotations)+} | [-unique)'-]{+unique+} {+ | if length != 1 then+} {+ error("unexpected number of manifests: " + length)+} {+ else . end+} {+ )+} {+'+} temp/index.json > temp/index.json.new mv temp/index.json.new temp/index.json {+# </build>+} {+# <push>+} crane push temp 'oisupport/staging-amd64:4b199ac326c74b3058a147e14f553af9e8e1659abc29bd3e82c9c9807b66ee43' rm -rf temp[-temp.tar-] # </push> # docker:24.0.7-windowsservercore-ltsc2022 [windows-amd64] @@ -20,7 +54,21 @@ docker pull 'mcr.microsoft.com/windows/servercore:ltsc2022@sha256:d4ab2dd7d3d0fc docker tag 'mcr.microsoft.com/windows/servercore:ltsc2022@sha256:d4ab2dd7d3d0fce6edc5df459565a4c96bbb1d0148065b215ab5ddcab1e42eb4' 'mcr.microsoft.com/windows/servercore:ltsc2022' # </pull> # <build> SOURCE_DATE_EPOCH=1700741054 {+\+} DOCKER_BUILDKIT=0 {+\+} docker build {+\+} --tag 'docker:24.0.7-windowsservercore-ltsc2022' {+\+} --tag 'docker:24.0-windowsservercore-ltsc2022' {+\+} --tag 'docker:24-windowsservercore-ltsc2022' {+\+} --tag 'docker:windowsservercore-ltsc2022' {+\+} --tag 'docker:24.0.7-windowsservercore' {+\+} --tag 'docker:24.0-windowsservercore' {+\+} --tag 'docker:24-windowsservercore' {+\+} --tag 'docker:windowsservercore' {+\+} --tag 'oisupport/staging-windows-amd64:9b405cfa5b88ba65121aabdb95ae90fd2e1fee7582174de82ae861613ae3072e' {+\+} --platform 'windows/amd64' {+\+} --file 'Dockerfile' {+\+} 'https://github.com/docker-library/docker.git#6d541d27b5dd12639e5a33a675ebca04d3837d74:24/windows/windowsservercore-ltsc2022' # </build> # <push> docker push 'oisupport/staging-windows-amd64:9b405cfa5b88ba65121aabdb95ae90fd2e1fee7582174de82ae861613ae3072e'

whalelines · 2024-01-26T18:02:16Z

.test/example-commands.sh

+rm temp.tar
+jq '
+	.manifests |= (
+		del(.[].annotations)


Why are the annotations deleted?

https://github.com/docker-library/meta-scripts/pull/17/files#r1467971216 👍

Then why do we add them in the build? (Sorry for my ignorance)

So that they show up in the provenance data 😅

whalelines · 2024-01-26T18:05:14Z

meta.jq

+				"mkdir temp",
+				"tar -xvf temp.tar -C temp",
+				"rm temp.tar",
+				# munge the index to what crane wants ("Error: layout contains 5 entries, consider --index")


Perhaps something to do with this?

Correct -- because of the limitations of an OCI layout as a docker save format, the output from docker buildx build --output type=tar generates an index.json with a new entry for every --tag flag:

$ docker buildx build --builder foo --quiet --output type=oci - <<<$'FROM scratch\nCMD []' --tag foo --tag bar --tag baz | tar --extract --to-stdout index.json | jq { "schemaVersion": 2, "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:1be03c71b27e7950b1a52e029ad139847a0771de9d2705ae8ab3742cd7d91ad6", "size": 288, "annotations": { "io.containerd.image.name": "docker.io/library/foo:latest", "org.opencontainers.image.created": "2024-01-26T18:48:33Z", "org.opencontainers.image.ref.name": "latest" }, "platform": { "architecture": "amd64", "os": "linux" } }, { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:1be03c71b27e7950b1a52e029ad139847a0771de9d2705ae8ab3742cd7d91ad6", "size": 288, "annotations": { "io.containerd.image.name": "docker.io/library/bar:latest", "org.opencontainers.image.created": "2024-01-26T18:48:33Z", "org.opencontainers.image.ref.name": "latest" }, "platform": { "architecture": "amd64", "os": "linux" } }, { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:1be03c71b27e7950b1a52e029ad139847a0771de9d2705ae8ab3742cd7d91ad6", "size": 288, "annotations": { "io.containerd.image.name": "docker.io/library/baz:latest", "org.opencontainers.image.created": "2024-01-26T18:48:33Z", "org.opencontainers.image.ref.name": "latest" }, "platform": { "architecture": "amd64", "os": "linux" } } ] }

Because the annotations are where the unique tags are stored, if we delete those, then unique can give us a single entry instead:

$ docker buildx build --builder foo --quiet --output type=oci - <<<$'FROM scratch\nCMD []' --tag foo --tag bar --tag baz | tar --extract --to-stdout index.json | jq '.manifests |= (del(.[].annotations) | unique)' { "schemaVersion": 2, "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:1be03c71b27e7950b1a52e029ad139847a0771de9d2705ae8ab3742cd7d91ad6", "size": 288, "platform": { "architecture": "amd64", "os": "linux" } } ] }

And now again, but this time with --provenance turned on (so we get an image index instead of an image manifest -- note the mediaType field in the descriptor):

$ docker buildx build --builder foo --provenance=mode=max --no-cache --quiet --output type=oci - <<<$'FROM hello-world' --tag foo --tag bar --tag baz | tar --extract --to-stdout index.json | jq '.manifests |= (del(.[].annotations) | unique)' { "schemaVersion": 2, "manifests": [ { "mediaType": "application/vnd.oci.image.index.v1+json", "digest": "sha256:a3cd4b5d34fcc7fe9ae1f6a9fd5459b7297e6cee7f6a5b200b715932a0fcb246", "size": 855 } ] }

tianon requested a review from yosifkit as a code owner January 25, 2024 21:31

tianon force-pushed the meta-refactor branch from bdc5f4d to aebff1b Compare January 25, 2024 22:46

tianon force-pushed the meta-refactor branch from aebff1b to 20200b6 Compare January 25, 2024 23:16

tianon mentioned this pull request Jan 25, 2024

Add "example" generated commands (so changes to generated commands are easier to see) #18

Merged

tianon force-pushed the meta-refactor branch from 20200b6 to 50d015e Compare January 25, 2024 23:40

tianon force-pushed the meta-refactor branch from 50d015e to af7b858 Compare January 25, 2024 23:47

tianon commented Jan 25, 2024

View reviewed changes

yosifkit approved these changes Jan 26, 2024

View reviewed changes

yosifkit merged commit a80a2c8 into docker-library:main Jan 26, 2024
1 check passed

yosifkit deleted the meta-refactor branch January 26, 2024 00:21

tianon mentioned this pull request Jan 26, 2024

Set org.opencontainers.image.base.name (and org.opencontainers.image.base.digest) appropriately #19

Merged

whalelines reviewed Jan 26, 2024

View reviewed changes

tianon mentioned this pull request Jan 30, 2024

Is it possible to detect if a tag was rebuilt due to a Dockerfile update? docker-library/faq#34

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor `meta.jq` to prepare for signing and `oci-import` implementation #17

Refactor `meta.jq` to prepare for signing and `oci-import` implementation #17

tianon commented Jan 25, 2024 •

edited

Loading

tianon commented Jan 25, 2024 •

edited

Loading

tianon commented Jan 25, 2024

tianon commented Jan 25, 2024

tianon Jan 25, 2024

whalelines Jan 26, 2024

tianon Jan 26, 2024

whalelines Jan 26, 2024 •

edited

Loading

tianon Jan 26, 2024

whalelines Jan 26, 2024

tianon Jan 26, 2024 •

edited

Loading

Refactor meta.jq to prepare for signing and oci-import implementation #17

Refactor meta.jq to prepare for signing and oci-import implementation #17

Conversation

tianon commented Jan 25, 2024 • edited Loading

tianon commented Jan 25, 2024 • edited Loading

tianon commented Jan 25, 2024

tianon commented Jan 25, 2024

tianon Jan 25, 2024

Choose a reason for hiding this comment

whalelines Jan 26, 2024

Choose a reason for hiding this comment

tianon Jan 26, 2024

Choose a reason for hiding this comment

whalelines Jan 26, 2024 • edited Loading

Choose a reason for hiding this comment

tianon Jan 26, 2024

Choose a reason for hiding this comment

whalelines Jan 26, 2024

Choose a reason for hiding this comment

tianon Jan 26, 2024 • edited Loading

Choose a reason for hiding this comment

Refactor `meta.jq` to prepare for signing and `oci-import` implementation #17

Refactor `meta.jq` to prepare for signing and `oci-import` implementation #17

tianon commented Jan 25, 2024 •

edited

Loading

tianon commented Jan 25, 2024 •

edited

Loading

whalelines Jan 26, 2024 •

edited

Loading

tianon Jan 26, 2024 •

edited

Loading