fix: pull base runtime image if it doesnt exist

[bchampp]

Which issue(s) does this change fix?

Why is this change necessary?

This change fixes a bug in SAM CLI where we are always pulling the base image (and setting the force_image_build flag to True) unnecessarily. This only happens if you don't already have the base image in your registry.

Explanation

SAM CLI uses the base image of the given runtime when building the rapid images for the functions runtime.

The generated Dockerfile looks something like this:

FROM public.ecr.aws/lambda/nodejs:22-x86_64
ADD aws-lambda-rie-x86_64 /var/rapid/                                                                                                
RUN mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie   

Note: This will look different if you are looking layers, this is just an example.

Then, SAM CLI makes an API call to docker.api.build(): https://github.com/aws/aws-sam-cli/blob/develop/samcli/local/docker/lambda_image.py#L424-L432

I think we were relying on an implicit behaviour from the docker client to keep a copy of the base image runtime locally after this call is made. However, this isn't the case - resulting in the bug.

You can recreate this locally by doing the following;

1. Create a simple Dockerfile

FROM public.ecr.aws/lambda/nodejs:22-x86_64
RUN echo "Building rapid image"

2. Build it:

docker build -f /tmp/test-dockerfile -t test-rapid /tmp

3. Check the images available in the registry:

docker images | grep -E "(nodejs|test-rapid)"

You'll observe that only the test-rapid image is persisted after the docker build, NOT the base nodejs runtime image.

How does it address the issue?

Now, when we do the comparison between the local and remote digest for the base image, we'll pull the base image locally. This avoids an implicit dependency on the docker client doing this for us, ensuring that the image always exists locally if it didn't exist the first time.

What side effects does this change have?

None. Customers will no longer see their image rebuilding on every sam local invoke.

Mandatory Checklist

PRs will only be reviewed after checklist is complete

• Review the generative AI contribution guidelines
• Add input/output type hints to new functions/methods
• Write design document if needed (Do I need to write a design document?)
• Write/update unit tests
• Write/update integration tests
• Write/update functional tests if needed
• make pr passes
• make update-reproducible-reqs if dependencies were changed
• Write documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[diegotry]

thanks for the improvement!

[bnusunny]

@bchampp can you share what sam cli command already rebuild the rapid images? From a quick test using sam local invoke, the rapid image is not always rebuilt.

➜  nodejs-zip-helloworld docker system prune -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: public.ecr.aws/lambda/nodejs:22-x86_64
untagged: public.ecr.aws/lambda/nodejs@sha256:1a4d6b8c37e1f8808b1ea433ac74d2905ae42e54b9b7b085b37a715563e12e7e
untagged: public.ecr.aws/lambda/nodejs:22-rapid-x86_64
deleted: sha256:f5fc091316aeb9c645b606aeddb871ebb517b983ec76f5ec1dd65a2f4e0d80b0
deleted: sha256:3581eb255d445342db6e13a1aee21f99a3b7387f89651082edbb16c66b9fcbc6
deleted: sha256:d5443285951af3727e798f3f88bb3ee1ecd8809b9e2f7926ba7a8d2da4f477c5
deleted: sha256:5ac95d46290ace89ed000c9e84e13e4e2c1654ac0cd75e2c496eaefabf41dffa
deleted: sha256:7f49c0f91a904dbee4476d55f75f3a2ff8eb12539a754705cb4387e0263a829a
deleted: sha256:80dfd58e9be279e5e7f4190f80294d26e4bf55662e1486608a6bb27b9dabea7b
deleted: sha256:c53e4f237aa76f14c90d22f50b0f1ee4bbad23dc0625de0fdb533ccb3ee75641
deleted: sha256:b2c94290329dc4acd18bda45d479455182690fe2e38043031a8b14ca2fd2e866
deleted: sha256:c4336072a1c923b29e93d73d9d222848533f2a7abb6a61f3615a2202849a1b59
deleted: sha256:f1739859ac724c1a3f26b23b021839da6dfafb464492e132fdc316dc25ac6ca9
deleted: sha256:13fa213d4d56b7f5da82b053e20d82f425a33664631e836582433a770a849342

Total reclaimed space: 434.7MB
➜  nodejs-zip-helloworld docker images -a      
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
➜  nodejs-zip-helloworld samdev local invoke   
2025-11-24 18:12:05 Attaching import module proxy for analyzing dynamic imports
Invoking app.lambdaHandler (nodejs22.x)                                                                                                                                                                                                                                                                               
Local image was not found.                                                                                                                                                                                                                                                                                            
Removing rapid images for repo public.ecr.aws/sam/emulation-nodejs22.x                                                                                                                                                                                                                                                
Building image.......................................................................................................
Using local image: public.ecr.aws/lambda/nodejs:22-rapid-x86_64.                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                      
Mounting /workplace/sunhua/github/bnusunny/aws-sam-projects/nodejs-zip-helloworld/hello-world as /var/task:ro,delegated, inside runtime container                                                                                                                                                                     
SAM_CONTAINER_ID: 7d0a9717b8722fa64f44a40898094fb98aab04573e49ef6740c0b62fc37067be                                                                                                                                                                                                                                    
START RequestId: 4d2b599b-203b-46c8-9b20-77f1bc190a49 Version: $LATEST
END RequestId: 2c97e150-7b1a-4631-9993-49094e7ece57
REPORT RequestId: 2c97e150-7b1a-4631-9993-49094e7ece57  Init Duration: 0.02 ms  Duration: 40.57 ms      Billed Duration: 41 ms  Memory Size: 128 MB     Max Memory Used: 128 MB
{"statusCode": 200, "body": "{\"message\":\"hello world\"}"}
[ERROR] [1764007938141] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
➜  nodejs-zip-helloworld docker images -a   
REPOSITORY                     TAG               IMAGE ID       CREATED         SIZE
public.ecr.aws/lambda/nodejs   22-rapid-x86_64   af44a99f96de   5 seconds ago   435MB
<none>                         <none>            e88b8495b8f7   9 seconds ago   428MB
public.ecr.aws/lambda/nodejs   22-x86_64         7f49c0f91a90   29 hours ago    414MB
➜  nodejs-zip-helloworld samdev local invoke
2025-11-24 18:12:27 Attaching import module proxy for analyzing dynamic imports
Invoking app.lambdaHandler (nodejs22.x)                                                                                                                                                                                                                                                                               
Local image is up-to-date                                                                                                                                                                                                                                                                                             
Using local image: public.ecr.aws/lambda/nodejs:22-rapid-x86_64.                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                      
Mounting /workplace/sunhua/github/bnusunny/aws-sam-projects/nodejs-zip-helloworld/hello-world as /var/task:ro,delegated, inside runtime container                                                                                                                                                                     
SAM_CONTAINER_ID: 784fda5d4ecb8b5c337b3592c0c86173bd3d97791fb318138a49f0b7698564f5                                                                                                                                                                                                                                    
START RequestId: 1ec114dd-92f5-4473-8bbb-364dc1cd5de1 Version: $LATEST
END RequestId: 3d26b4f6-1f1f-40c4-a770-23af88e10aa1
REPORT RequestId: 3d26b4f6-1f1f-40c4-a770-23af88e10aa1  Init Duration: 0.02 ms  Duration: 40.17 ms      Billed Duration: 41 ms  Memory Size: 128 MB     Max Memory Used: 128 MB
{"statusCode": 200, "body": "{\"message\":\"hello world\"}"}
[ERROR] [1764007950309] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
➜  nodejs-zip-helloworld samdev local invoke
2025-11-24 18:12:38 Attaching import module proxy for analyzing dynamic imports
Invoking app.lambdaHandler (nodejs22.x)                                                                                                                                                                                                                                                                               
Local image is up-to-date                                                                                                                                                                                                                                                                                             
Using local image: public.ecr.aws/lambda/nodejs:22-rapid-x86_64.                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                      
Mounting /workplace/sunhua/github/bnusunny/aws-sam-projects/nodejs-zip-helloworld/hello-world as /var/task:ro,delegated, inside runtime container                                                                                                                                                                     
SAM_CONTAINER_ID: 066ceffd115bd301923af35869e5a823c353edc3c1f7e3fcb8cdb14153c3668d                                                                                                                                                                                                                                    
START RequestId: 2e76704e-c90e-4060-bdeb-f2b6c9bd38ae Version: $LATEST
END RequestId: b8eee126-f758-4fba-89b1-40e58073d9fa
REPORT RequestId: b8eee126-f758-4fba-89b1-40e58073d9fa  Init Duration: 0.02 ms  Duration: 40.68 ms      Billed Duration: 41 ms  Memory Size: 128 MB     Max Memory Used: 128 MB
{"statusCode": 200, "body": "{\"message\":\"hello world\"}"}
[ERROR] [1764007962120] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint

[bnusunny]

SAM CLI uses docker-py library to build docker images. It uses the legacy builder, which does pull down the base images and we can see the them using docker images.

Newer docker CLI uses buildkit as default builder. Buildkit does pull down the base images into its build cache. But the base images do not appear as a separate entry in docker images.

[bchampp]

It uses the legacy builder, which does pull down the base images and we can see the them using docker images.

I think this means the test I added to recreate it using the Docker CLI directly isn't accurate because of this. @valerena showed me that using the Docker CLI, the base image gets cached instead of appearing in the docker images output, making subsequent build times much faster. However, as @bnusunny points out, the docker client the SAM CLI is using has a different behaviour which is expected to keep a copy of the base image directly.

This doesn't seem to be happening for me anymore, so I'm closing the PR. I'm not sure why I was seeing this happening often last week, but I'm chalking it up to something on my end. I think there might be value in us explicitly modelling this behaviour, but I think it would be a breaking change if the docker client changed their behaviour here.