OCI layout extensions #2633
OCI layout extensions #2633
Conversation
Images in the index can now be referenced via the @sourceIndex syntax. Signed-off-by: Miloslav Trmač <mitr@redhat.com> Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
The new API allows for listing all manifests in an OCI layout's index. Signed-off-by: Miloslav Trmač <mitr@redhat.com> Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
On Linux machines, the file will not be copied but hardlinked. Signed-off-by: Miloslav Trmač <mitr@redhat.com> Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
|
In draft for now. |
| // It computes, and returns, the digest and size of the used file. | ||
| // | ||
| // This function can be used instead of dest.PutBlob() where the ImageDestination requires PutBlob() to be called. | ||
| func PutBlobFromLocalFile(ctx context.Context, dest types.ImageDestination, file string) (digest.Digest, int64, error) { |
There was a problem hiding this comment.
There was a problem hiding this comment.
Ah, scratch that. The caller is expected to update the manifest in those cases.
I implemented a very similar mechanism in buildah source add which automatically adds the layers to the manifest and updates the digests etc:
https://github.com/containers/buildah/blob/main/internal/source/add.go#L46
I am OK with the current design but it's probably something we should add to the docs of the function.
There was a problem hiding this comment.
Ah, scratch that. The caller is expected to update the manifest in those cases.
Yes. Copy&pasting from #2567 (comment):
The way the ImageDestination API works, PutBlob doesn’t typically need to worry about the MIME type: the MIME type comes via PutManifest.
While it is true that the PutBlobWithOptions gets a full types.BlobInfo, and
Line 852 in 0c42f2f
Line 168 in 0c42f2f
PutBlobOptions.IsConfig).
Also, specifically in the OCI transport, the PutBlob* code really doesn’t care about the MIME type. It is going to create a sha256-named file, and as far as the layout definition is concerned, it’s perfectly valid to refer to a single blob in two different manifests using two different MIME types. So, for this OCI-transport-specific function, I don’t think adding a MIME type is right.
| os.Remove(blobFile.Name()) | ||
| if err := os.Link(file, blobFile.Name()); err != nil { |
There was a problem hiding this comment.
This is conceptually racy against the CreateTemp above. An alternative could be to create a temporary directory, and to create this link to a file in that temporary directory.
| os.Remove(blobFile.Name()) | ||
| if err := os.Link(file, blobFile.Name()); err != nil { |
There was a problem hiding this comment.
I’m EDIT not quite sure about a hard-link to the user’s input; the user could continue to edit that file afterwards. At the very least this would need a strong API documentation.
I think a reflink would be uncontroversial, OTOH not always possible.
There was a problem hiding this comment.
Fair point. Reflinks work on Linux (xfs, btrfs) and on Mac OS (? see unix package). Hardlinks work everywhere, so they are more portable.
If we envision podman artifact having to deal with huge files (e.g., LLMs), then hardlinks seem more attractive. On the other hand, any change to the file after a podman artifact add would corrupt the image as the digest wouldn't match anymore. The digest could be recomputed on artifact push for instance.
OR will podman artifact on the remote-clients be talking to podman-machine? In that case, the reflinks would only work on Linux.
@baude can you give direction?
|
|
||
| // need to explicitly close the file, since a rename won't otherwise work on Windows | ||
| blobFile.Close() | ||
| if err := os.Rename(blobFile.Name(), blobPath); err != nil { |
There was a problem hiding this comment.
We might already have a file with exactly that digest; that’s not an error.
There was a problem hiding this comment.
Will add a test for that case 👍
Taken over from #2567 but: