Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ai_cd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: superfly/flyctl-actions/setup-flyctl@master
- run: flyctl deploy --config apps/ai/fly.toml --dockerfile apps/ai/Dockerfile --remote-only --build-arg APP_VERSION=${{ needs.compute-version.outputs.version }}
- run: flyctl deploy --config apps/ai/fly.toml --dockerfile apps/ai/Dockerfile --remote-only --build-arg APP_VERSION=${{ needs.compute-version.outputs.version }} --build-arg GIT_SHA=${{ github.sha }}
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

Expand Down
4 changes: 3 additions & 1 deletion apps/ai/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,13 @@ RUN cargo chef prepare --recipe-path recipe.json

FROM rust:${RUST_VERSION}-bookworm AS build
ARG APP_VERSION
ARG GIT_SHA
RUN apt-get update && apt-get install -y --no-install-recommends pkg-config libssl-dev ca-certificates && rm -rf /var/lib/apt/lists/*
RUN cargo install cargo-chef sccache --locked
ENV RUSTC_WRAPPER=sccache \
SCCACHE_DIR=/sccache \
APP_VERSION=${APP_VERSION}
APP_VERSION=${APP_VERSION} \
VERGEN_GIT_SHA=${GIT_SHA}
Comment on lines 21 to +24
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 VERGEN_GIT_SHA in ENV before cargo chef cook invalidates dependency cache on every deploy

Setting VERGEN_GIT_SHA=${GIT_SHA} in the ENV instruction at line 24 causes Docker layer cache invalidation for all subsequent layers, including the cargo chef cook step (lines 27-30). Since GIT_SHA changes on every single deploy, the dependency compilation cache is busted every time, defeating the purpose of cargo-chef.

Root Cause and Impact

Docker layer caching works by checking if each instruction and its inputs are identical to a previous build. The ENV instruction at lines 21-24 includes VERGEN_GIT_SHA=${GIT_SHA}, which changes with every commit. This means every layer after line 24 is rebuilt from scratch, including:

  • Line 27-30: cargo chef cook (dependency compilation — the most expensive step)
  • Line 35-38: cargo build (application compilation)

The whole point of the cargo-chef pattern is to separate dependency compilation from application compilation so dependencies are cached. This change negates that benefit.

The VERGEN_GIT_SHA env var is only needed during cargo build (line 38), not during cargo chef cook. The fix is to move the VERGEN_GIT_SHA env var to after the cargo chef cook step, e.g., by splitting the ENV instruction or adding a separate ENV VERGEN_GIT_SHA=${GIT_SHA} after line 30.

Note: APP_VERSION has the same pre-existing issue, but it changes less frequently (only on version bumps), whereas GIT_SHA changes on every deploy.

Impact: Every deploy will recompile all dependencies from scratch (layer cache miss), significantly increasing build times (potentially 10-30+ minutes of unnecessary compilation on each deploy).

Prompt for agents
In apps/ai/Dockerfile, split the ENV instruction so that VERGEN_GIT_SHA (and ideally APP_VERSION too) is set AFTER the cargo chef cook step but BEFORE the cargo build step. This preserves Docker layer caching for dependency compilation.

Specifically:
1. Lines 21-24: Change the ENV to only set RUSTC_WRAPPER and SCCACHE_DIR (the values that don't change between deploys):
   ENV RUSTC_WRAPPER=sccache \
       SCCACHE_DIR=/sccache

2. After line 30 (after cargo chef cook), add a new ENV instruction for the deploy-specific variables:
   ENV APP_VERSION=${APP_VERSION} \
       VERGEN_GIT_SHA=${GIT_SHA}

This way, the cargo chef cook layer is only invalidated when dependencies change, not on every deploy.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

WORKDIR /app
COPY --from=planner /app/recipe.json recipe.json
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=locked \
Expand Down