Capture and elevate cloud executor injected metadata, specifically the Docker image digest

[tfallon-ionis]

New feature

Hi, we're using Nextflow on AWS Batch.

We find all our tasks have this environmental variable injected ECS_CONTAINER_METADATA_URI_V4, that contains an HTTP+JSON endpoint to GET useful metadata from:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html

And we can then call the endpoint and get a json of really handy metadata:
curl -sS --fail --connect-timeout 5 --max-time 10 "\$ECS_CONTAINER_METADATA_URI_V4" > ecs_metadata.json || true ## variable is an endpoint injected into all AWS Batch jobs that run on ECS-backed compute environments

Most importantly is the ImageID (i.e. content based digest) of the Docker container that is running, that I haven't seen accessible anywhere else in Nextflow
cat ecs_metadata.json | jq '.ImageID' "sha256:47d74d2f1d360a3167ea062129a4af229af095ef0fd23b842f62647e3ad29c6c"

The new feature would be to bake in fetching and parsing this metadata as appropriate for Nextflow cloud executors like AWS Batch.

Use case

Whenever you're running Docker containers on the cloud, and you want traceability as to the source container. (Other metadata presumably useful as well, but this is our use case with ImageID).

Suggested implementation

For each cloud executor, research if they have a similar pattern to this AWS Batch pattern, and then make a bespoke before script execution. Then injest it and make it available via trace.csv , report.html, etc.

[tfallon-ionis]

Okay, an update on a bit of unexpected AWS behavior.

It turns out that calling ECS_CONTAINER_METADATA_URI_V4 returns a sha256: digest, which is not the same as, but is related to (I believe still in an unmutable, content-based digest way) the preferred image digest shown on ECR through the AWS Console and by Docker Desktop docker inspect of the image pulled from ECR. I believe this ECS container metadata ImageID is more accurately called the config.digest.

The below incantation returns the preferred image digest from the config.digest and the image tag (that is also available from ECS_CONTAINER_METADATA_URI_V4):

export config_digest='sha256:a0b26893a26fc47585711dd6c71c8afea3fd4c1c09a681acd874be3c2b2d7afa'
export tag='<REDACTED>'
export region='<REDACTED>'
export registry_id='<REDACTED>'
export repository_name='<REDACTED>'

manifest_digest="$(
  aws ecr batch-get-image \
    --region "$region" \
    --registry-id "$registry_id" \
    --repository-name "$repository_name" \
    --image-ids imageTag="$tag" \
    --output json \
  | jq -r --arg config_digest "$config_digest" '
      .images[]
      | select((.imageManifest | fromjson | .config.digest) == $config_digest)
      | .imageId.imageDigest
    '
)"

if [ -n "$manifest_digest" ]; then
  aws ecr describe-images \
    --region "$region" \
    --registry-id "$registry_id" \
    --repository-name "$repository_name" \
    --image-ids imageDigest="$manifest_digest" \
    --output json \
  | jq -r '
      .imageDetails[]
      | [.imageDigest, (.imageTags // [] | join(","))]
      | @tsv
    '
fi

sha256:ff93ff3bdc6911f64ac3cf4b292280dcf01876c199a70c894e8388d9405c50ac