Buildpacks publish fix

[casibbald]

Buildpacks: set Publish when pushing to avoid daemon export on containerd (e.g. Mac)

Related: buildpacks/pack#2272 (pack build 27× slower / digest errors with containerd image store), buildpacks/pack#2270 (build fails with containerd + untrusted builder – digest not found)

Description

When users run skaffold build --push (or equivalent) with the buildpacks builder, Skaffold invokes the pack library with pack.BuildOptions. Previously, Skaffold never set Publish: true, so pack always wrote the built app image to the Docker daemon first; Skaffold would then tag and push. On daemons that use containerd storage (Docker Desktop and Colima on Mac, default for new Docker Desktop installs), that export path can:

• Fail with digest errors (e.g. unable to create manifests file: NotFound: content digest sha256:...: not found – see pack#2270)
• Or be severely degraded (~9s export, ~20s delay before Analyzing – see pack#2272)

Setting Publish: b.pushImages in pack.BuildOptions when building ensures that when Skaffold is doing a push build, pack publishes the image directly to the registry and does not write it to the daemon. That avoids the containerd digest path entirely.

When pushImages is true, we also return the requested tag immediately after the build and do not call localDocker.Tag or localDocker.Push, because pack has already pushed the image.

Changes

1. pkg/skaffold/build/buildpacks/lifecycle.go
   In the pack.BuildOptions passed to runPackBuildFunc, set Publish: b.pushImages.

2. pkg/skaffold/build/buildpacks/build.go
   In Build(), when b.pushImages is true, return tag, nil and skip the usual localDocker.Tag / localDocker.Push path.

User facing changes

• Before: skaffold build --push with buildpacks on Mac (containerd-backed daemon) could fail with a digest error or hang/slow during export; users had to use workarounds (e.g. run pack with --publish outside Skaffold).
• After: skaffold build --push with buildpacks instructs pack to publish directly to the registry. Builds complete successfully on containerd-backed daemons without writing the app image to the daemon.

For local registries (e.g. localhost:5001), the pack lifecycle runs inside a container where localhost is the container itself. Users should set the default repo to a host-reachable address (e.g. --default-repo host.docker.internal:5001 or SKAFFOLD_DEFAULT_REPO=host.docker.internal:5001) so the lifecycle can push.

Follow-up Work

• Consider documenting in Skaffold docs that for local registry on Mac, --default-repo host.docker.internal:5001 may be required for buildpacks push.

[gemini-code-assist]

Summary of Changes

Hello @casibbald, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the reliability and performance of Skaffold's buildpacks builder when pushing images, particularly for users on macOS with containerd-backed Docker daemons. By instructing the underlying pack library to publish images directly to the registry during a push operation, it eliminates previous issues such as digest errors and slow export times. This change streamlines the developer experience by ensuring a more robust and efficient build-and-push workflow for buildpack-based applications.

Highlights

• Improved Buildpacks Push Reliability: Addresses and resolves issues where skaffold build --push with buildpacks would fail with digest errors or experience severe performance degradation on Docker daemons using containerd storage (common on macOS).
• Direct Image Publishing: Configures the pack library to directly publish the built image to the registry when skaffold build --push is active, bypassing the problematic Docker daemon export path.
• Optimized Post-Build Workflow: Modifies the build process to skip redundant localDocker.Tag and localDocker.Push operations when pack has already handled the direct publishing to the registry.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• PR_DESCRIPTION.md
  • Added a new file containing the detailed pull request description, outlining the problem, solution, and user-facing changes.
• pkg/skaffold/build/buildpacks/build.go
  • Modified the Build function to conditionally return the image tag immediately if b.pushImages is true, effectively skipping the localDocker.Tag and localDocker.Push calls when direct publishing is enabled.
• pkg/skaffold/build/buildpacks/lifecycle.go
  • Updated the build function to set the Publish field in pack.BuildOptions to b.pushImages. This ensures that the pack library publishes the image directly to the registry when a push operation is intended.

Activity

• No human activity (comments, reviews) has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request aims to fix an issue with buildpacks on containerd-based systems by having pack publish images directly to the registry. The changes correctly delegate the push operation to pack and adjust Skaffold's logic to avoid tagging and pushing the image itself. However, a critical issue has been identified: while pack is instructed to publish the image, it's consistently using the :latest tag instead of the unique tag generated by Skaffold. This logic flaw means the intended image tag is not pushed to the registry, leading to deployments of stale or non-existent images. This poses a high security risk as it can result in the deployment of vulnerable code versions and will cause deployments to fail with "image not found" errors. The build process should be updated to ensure that the intended tag is correctly published to the registry when direct publishing is enabled. A detailed comment with a suggested fix has been left in pkg/skaffold/build/buildpacks/lifecycle.go.

[gemini-code-assist]

When Publish: true is set, pack will publish the image specified in the Image option. Currently, this is always ...:latest (from line 68 and 108). This means the uniquely tagged image (e.g., with a git commit SHA) is never pushed to the registry. Instead, only the :latest tag is pushed.

The Build function in pkg/skaffold/build/buildpacks/build.go then returns the unique tag, assuming it has been pushed. This will lead to "image not found" errors downstream when Kubernetes tries to pull the uniquely tagged image.

When publishing, pack should be instructed to build and push the final, unique tag. The :latest tag should only be used for local, non-pushed builds to leverage caching.

I suggest changing the image passed to pack based on whether we are pushing or not. Since I cannot suggest changes outside the diff, here is an example of how to fix this in pkg/skaffold/build/buildpacks/lifecycle.go:

// ...
	parsed, err := docker.ParseReference(tag)
	if err != nil {
		return "", fmt.Errorf("parsing tag %q: %w", tag, err)
	}
	latest := parsed.BaseName + ":latest"

	imageToBuild := latest
	if b.pushImages {
		imageToBuild = tag
	}
// ...
	if err := runPackBuildFunc(ctx, out, b.localDocker, pack.BuildOptions{
		// ...
		Image:             imageToBuild,
		// ...
		Publish:           b.pushImages,
	}); err != nil {
// ...

The key is to conditionally set the Image option for pack.BuildOptions.

[gemini-code-assist]

The logic introduced here when b.pushImages is true creates a discrepancy between the image pushed to the registry and the tag returned by Skaffold.

In pkg/skaffold/build/buildpacks/lifecycle.go, the build process is configured to publish the image using the :latest tag (e.g., myrepo/myimage:latest). However, this Build function returns the original tag (e.g., myrepo/myimage:v1) and skips the local tagging and pushing steps.

As a result, the intended tag is never pushed to the registry. Subsequent steps in the Skaffold pipeline (like deployment) will attempt to use the intended tag, which will either fail or, more critically, pull a stale version of the image if it already existed in the registry. This can lead to the deployment of incorrect or outdated code, potentially bypassing security fixes included in the new build.