Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release-5.32 Zstd backports #2507

Closed
wants to merge 27 commits into from
Closed

release-5.32 Zstd backports #2507

wants to merge 27 commits into from

Commits on Aug 7, 2024

  1. Bump to 5.32.1-dev 

    Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    5af61e0 View commit details
    Browse the repository at this point in the history

  2. Compute the schema1 DiffID list based on m.LayerInfos() 

    We are already calling m.LayerInfos() anyway, so there is ~no
extra cost. And using LayerInfos means we don't need to worry
about reversing the order of layers, and we will have access
to the layer index, allowing us to acccess the indexTo* fields
in the future.

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    dca0610 View commit details
    Browse the repository at this point in the history

  3. Improve the documentation of layer identification 

    - Don't claim that we only use compressed digests.
- Explicitly document that we assume TOC digests to be unambiguous
- Actually define the term "DiffID".
- Be more precise in computeID about the criteria being layer identity,
  not where we pull the layer from.

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    437b7fb View commit details
    Browse the repository at this point in the history

  4. Allow returning (and reporting) unexpected errors from computeID 

    Some errors are severe enough that just logging and continuing is
not really worthwhile.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    59f1890 View commit details
    Browse the repository at this point in the history

  5. Centralize collecting _consistent_ layer data into trustedLayerIdenti… 

    …tyDataLocked

Currrently we "only" have indexToTOCDigest and blobDiffIDs, but we will make this
more complex.

Centralizing the consumption of these fields into trustedLayerIdentityDataLocked
ensure that all consumers interpret the data exactly consistently (and it also
allows us to use a single "trusted" variable instead of 2/3 individual ones).

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    e405b01 View commit details
    Browse the repository at this point in the history

  6. Add TOC digest <-> uncompressed digest mapping to BIC 

    The new code is not called, so it should not change behavior
(apart from extending the BoltDB/SQLite schema).

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    3788220 View commit details
    Browse the repository at this point in the history

  7. Set up storage destination to support TOC-based layers identified in … 

    …storage by DiffID

If we can, prefer identifying layers by DiffID, because multiple TOCs can map to the
same DiffID; and because it maximizes reuse with non-TOC layers.

For now, the new situation is unreachable.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    ad6d40d View commit details
    Browse the repository at this point in the history

  8. Split reusedBlobFromLayerLookup from tryReusingBlobAsPending 

    We will add one more instance of this, so share the code.

Should not change behavior (it does remove one unreachable code path).

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    2c22da0 View commit details
    Browse the repository at this point in the history

  9. Use BlobInfoCache to share more layers if (TOC->Uncompressed) mapping… 

    … is known

- Multiple TOC values might correspond to a single DiffID (e.g. if different
  compression levels are used); try to share them all, identified by DiffID
  (so that we also reuse with non-TOC pulls).
  - LayersByTOCDigest only uses a single TOC digest per layer; BlobInfoCache
    allows multiple matches, matches layers which have been since deleted,
    and potentially matches TOC digests which we have created by pushing
    but haven't pulled yet.
- On reuse, we can now use DiffID-based layer identities even if the reuse
  was TOC~driven.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    cc25f17 View commit details
    Browse the repository at this point in the history

  10. Record the (TOC digest, uncompressed digest) data when we compress la… 

    …yers

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    0e79045 View commit details
    Browse the repository at this point in the history

  11. Use the uncompressed digest we got from a BlobInfoCache for chunked l… 

    …ayers

- Rely on it instead of triggering the "untrusted DiffID" logic
- Also propagate it to storage

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    874ad0e View commit details
    Browse the repository at this point in the history

  12. HACK: Don't compress with zstd:chunked when encrypting 

    Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    072a576 View commit details
    Browse the repository at this point in the history

  13. Fix data returned when returning uncompressed data on a c/storage lay… 

    …er match

The rules expect us to set manifest editing updates.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    7292f9a View commit details
    Browse the repository at this point in the history

  14. Turn CandidateCompression into CandidateTemplateWithCompression 

    ... and add CandidateWithLocation and CandidateWithUnknownLocation ,
so that the BIC implementations only need to deal with one value
instead of carrying around three; we will want to add one more,
and encapsulating them all into a single template will make it
transparent to the cache implementations.

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    6e33bdb View commit details
    Browse the repository at this point in the history

  15. Make the fields of CandidateWithTime private 

    ... just because we now can, and to nudge all future caches to be designed
around CandidateTemplateWithCompression.

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    d816552 View commit details
    Browse the repository at this point in the history

  16. Reformat CandidateTemplateWithCompression a bit 

    We will add more logic to the default case, so sharing the
CandidateCompressionMatchesReuseConditions call is not going to be
as easy. Split the two code paths.

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    b36f716 View commit details
    Browse the repository at this point in the history

  17. Introduce blobinfocache.DigestCompressorData 

    We will want to record more than a single alghoritm name. For now,
just introduce the structure and modify users, we'll add the new fields
later.

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    1ff3519 View commit details
    Browse the repository at this point in the history

  18. Always record only the base variant information about consumed source… 

    … blobs

... because we don't trust the TOC data, if any.

This allows us to remove the zstd:chunked hack; we, at least,
now record those blobs as zstd.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    577b535 View commit details
    Browse the repository at this point in the history

  19. Unrelated: Fix a bug in SQLite BlobInfoCache 

    If we don't know an uncompressed digest, don't try using "".

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    4df6647 View commit details
    Browse the repository at this point in the history

  20. Fix a comment 

    Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    f4b5e90 View commit details
    Browse the repository at this point in the history

  21. Add digest -> specific variant, annotation data to BIC 

    The cache implementations are recording both the base and specific compression variant;
CandidateLocations2 all call CandidateTemplateWithCompression to choose the
appropriate variants to return based on CandidateLocations2Options.

This way, neither the BIC implementations nor the transports are not responsible for
converting zstd:chunked entries to zstd entries if the user wants the latter.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    08100b7 View commit details
    Browse the repository at this point in the history

  22. Record the specific variant, and TOC annotations, for blobs we compress 

    Introduce distinct uploadedCompressorBaseVariantName and
uploadedCompressorSpecificVariantName fields; that way we now never
call RecordDigestCompressorData with inconsistent zstd / zstd:chunked in one field,
so we can always record data when we see, or create, a zstd:chunked layer,
removing the current hack.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    c2d35c0 View commit details
    Browse the repository at this point in the history

  23. Extend private.ReusedBlob to allow zstd:chunked reuses 

    - Add a CompressionAnnotations field
- Allow turning a known-zstd blob into a zstd:chunked one if we
  know the right annotations

This just adds the fields, nothing sets them yet, should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    cced793 View commit details
    Browse the repository at this point in the history

  24. Allow dockerImageDestination to reuse zstd:chunked blobs 

    - Return the required annotations, if we have them
- If we have a zstd blob and the BIC contains the annotations,
  we don't check for the blob's presence initially. In that case,
  don't skip it if we find the BIC annotations.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    0800e84 View commit details
    Browse the repository at this point in the history

  25. Detect zstd:chunked format in source blobs 

    ... instead of only treating it as zstd.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    2fff234 View commit details
    Browse the repository at this point in the history

  26. Release 5.32.1 

    Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    a8aa8c4 View commit details
    Browse the repository at this point in the history

  27. Bump to 5.32.2-dev 

    Signed-off-by: Miloslav Trmač <mitr@redhat.com>
    @mtrmac
    mtrmac committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    dda9562 View commit details
    Browse the repository at this point in the history