Skip to content

Conversation

@mzihlmann
Copy link
Collaborator

@mzihlmann mzihlmann commented Jan 11, 2026

Fixes #440

Description
Kaniko is not designed to run multiple times in the same pod, so adding retries around its execution causes hard failures. While Kaniko provides a --cleanup option, it only removes the filesystem created during the build and only on success. This change extends the cleanup to also remove Kaniko’s working directory, which stores intermediate build output, and ensures cleanup happens even when the build fails.

The following new directories are removed:

  • DockerfilePath: The dockerfile we're building, it will be overwritten anyways
  • KanikoIntermediateStagesDir: Where we store stages as tarball/oci
  • BuildContextDir <- this was the root cause of the issue
  • KanikoInterStageDepsDir: Files that are copied between stages
  • KanikoLayersDir: Where we store tarball/oci snapshots of individial layers
  • KanikoSecretsDir: Where secrets are copied to, it would be wrong to reuse them

We specifically do not remove:

  • KanikoCacheDir: We might want to reuse cache dirs, but you could argue that this is dangerous too
  • KanikoSwapDir: This directory must be empty anyways
  • DockerConfigDir: This is not created by kaniko

It's a bit a debate which directories should be reusable, as it depends on whether we rebuild the same dockerfile, same target etc etc. I know that a handful of people build multiple targets in the same pod using this strategy, we eventually want to support that out of the box in a single run.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

  • Includes unit tests
  • Adds integration tests if needed.

See the contribution guide for more details.

Reviewer Notes

  • The code flow looks good.
  • Unit tests and or integration tests added.

Release Notes

Describe any changes here so maintainer can include it in the release notes, or delete this block.

Examples of user facing changes:
- kaniko adds a new flag `--registry-repo` to override registry

@mzihlmann
Copy link
Collaborator Author

reproducer repository here https://gitlab.com/martizih/kaniko-retry

@mzihlmann mzihlmann marked this pull request as ready for review January 11, 2026 12:43
@mzihlmann mzihlmann force-pushed the mz440-cleanup-kaniko-dir branch from 374e0a8 to 274401c Compare January 11, 2026 12:46
@mzihlmann
Copy link
Collaborator Author

The documentation was already unambiguous that the cleanup will happen on failure too, so that change is correct and I would consider implementation so far bogus.

@mzihlmann mzihlmann requested review from 0hlov3, BobDu, babs and nejch January 11, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cleanup does not seem to be empty

2 participants