I propose adding the support for layers inside of OCX
Why would it be beneficiary to support layers?
1. Further OCI compliance
Images originating from the OCI ecosystem could be processed more easily in the OCX ecosystem, and vice versa
2. Deduplication of common artifacts within different images
Compiler toolcontains potentially contain the same headers and libraries for multiple execution platforms, only a subset of the files (mostly the executables) are execution platform specific. An informed packager could split-up the set of inputs (e.g. the zipped toolchains of a given version for all supported platforms) and produce a set of images with the minimal layers required.
The impact of this optimization is, on the one hand, reduced memory usage in the registry itself and lower data transfer to any mirrors. And, on the other hand, reduced storage space usage and data transfer to environments that use a wide variety of different versions of the tool
Self-contained applications requiring a runtime environment e.g. Python, Java, c#, etc. could build their image by using an existing layer containing the runtime environment and adding their implementation artifacts into another layer via "structured packing".
The benefit of this optimization is, once again, reduced memory usage in the registry and lower data transfer to any mirrors. It also results in reduced memory usage and data transfer to environments that use a variety of different tools requiring the same runtime environment.
Implementation alternatives
a. fully OCI compliant layers - layers as changeset
Pro:
- oci compliance - well-known workflow and existing tooling for OCI should be usable for OCX too
- images could be easily adjusted by adding another layer
Contra:
- Layers have to be applied sequentially on each other, an upper layer could override a file from a lower layer.
- Difficult to parallelize the unpacking into a directory where the layers are merged again
- Alternative approach by merging the layer via vfs in overlayfs isn't portable
- Intensionally overriding files in higher layers open the door for untransparent / chaotically crafted packages
b. overlap free layers - file structures are merged together.
Files other than directories aren't allowed to be located at the same path
layer-gcc-arm64/
└── usr/
└── bin/
├── gcc
├── g++
└── cpp
layer-gcc-common/
└── usr/
└── lib/
└── gcc/
└── 13/
├── include-fixed/
├── specs
└── plugin/
Pro:
- easy to implement parallel unpacking - no support for whiteouts required
Contra
- This is not intuitively obvious if you're coming from the OCI world. Furthermore, standard OCI images are potentially incompatible. But not the other way around.
c. (completely) overlap free layers - file structures are merged together.
Files/directories aren't allowed to be located at the same path -> The file trees must differ at the image root level
Pro:
- layers must be unpacked only once, even if used by multiple images as files/directories could be hard- or symlinked in to image root easily
Contra
- When creating a deduplicated image, the package design options are significantly limited
- Would be even less intuitive, if you're coming from the OCI world.
layer-gcc-common/
└── gcc-common/
└── usr/
└── lib/
└── gcc/
└── 13/
├── specs
└── plugin/
layer-gcc-amd64/
└── gcc-amd64/
└── usr/
└── bin/
├── gcc
├── g++
└── cpp
Further notes
To make use of this feature easily and effectively, there should also be a tool that takes as input a set of images of the same package in different variations or versions and outputs new images with optimized / deduplicated layering structure.
I propose adding the support for layers inside of OCX
Why would it be beneficiary to support layers?
1. Further OCI compliance
Images originating from the OCI ecosystem could be processed more easily in the OCX ecosystem, and vice versa
2. Deduplication of common artifacts within different images
Compiler toolcontains potentially contain the same headers and libraries for multiple execution platforms, only a subset of the files (mostly the executables) are execution platform specific. An informed packager could split-up the set of inputs (e.g. the zipped toolchains of a given version for all supported platforms) and produce a set of images with the minimal layers required.
The impact of this optimization is, on the one hand, reduced memory usage in the registry itself and lower data transfer to any mirrors. And, on the other hand, reduced storage space usage and data transfer to environments that use a wide variety of different versions of the tool
Self-contained applications requiring a runtime environment e.g. Python, Java, c#, etc. could build their image by using an existing layer containing the runtime environment and adding their implementation artifacts into another layer via "structured packing".
The benefit of this optimization is, once again, reduced memory usage in the registry and lower data transfer to any mirrors. It also results in reduced memory usage and data transfer to environments that use a variety of different tools requiring the same runtime environment.
Implementation alternatives
a. fully OCI compliant layers - layers as changeset
Pro:
Contra:
b. overlap free layers - file structures are merged together.
Files other than directories aren't allowed to be located at the same path
Pro:
Contra
c. (completely) overlap free layers - file structures are merged together.
Files/directories aren't allowed to be located at the same path -> The file trees must differ at the image root level
Pro:
Contra
Further notes
To make use of this feature easily and effectively, there should also be a tool that takes as input a set of images of the same package in different variations or versions and outputs new images with optimized / deduplicated layering structure.