Static asset server

This repository contains a ahead-of-time static asset optimization pipeline that generates a container image providing a standalone static asset server.

Optimization pipeline

The optimization pipeline, whose responsibility is generating the optimized static assets as well as the index file, is implemented in the compress.sh script. This script relies on well-known utilities (e.g. brotli, zopfli, zstd, optipng, mozjpeg, cwebp, gifsicle, svgo, ...) to perform these tasks.

Image asset optimizations

Every source image is converted to variants of each image, each in a different format. Currently variants are only generated if the resulting file is smaller than that of the original image. This table shows the which variants are created for each source image type.

• ✅ means that the variant is created for that source image type (only if the resulting file is smaller than the source file)
• ✔️ means that the variant is created for that source image type (regardless if the resulting file is smaller than the source file)
• ⏳ means that generation of this variant is TODO

↓ Source / Variants →	JPEG	GIF	PNG	WebP	APNG	AVIF	HEIF	JPEG-XL	SVG
JPEG	✅			✅		✅	✅	✅
GIF	✅	✅	✅	✅	✅	✅	⏳	✅
GIF (animated)		✅		✅	✅	⏳	⏳	✅
PNG	✅	⏳	✅	✅	⏳	✅	✅	✅
WebP	✅	⏳	✅	⏳	⏳	✅	⏳	✅
WebP (animated)		⏳		⏳	⏳	⏳	⏳	⏳
APNG	⏳	⏳	⏳	⏳	⏳	⏳	⏳	⏳
AVIF	⏳	⏳	⏳	⏳	⏳	⏳	⏳	⏳
HEIF	⏳	⏳	⏳	⏳	⏳	⏳	⏳	⏳
JPEG-XL	⏳			⏳		⏳	⏳	⏳
SVG									✅

Notes:

• WebP → WebP is not ✅ because we don't perform any optimization, so the original file is used.

Other asset optimizations

• JSON files are minified using jq
• Javascript files are minified using UglifyJS

Finally, all compressible files (including e.g. SVG, JSON, Javascript and HTML) are statically compressed with zopfli (gzip), brotli and zstandard (zstd) at maximum compression (something that would be normally impractical if done on-the-fly).

Asset server

The standalone HTTP server is written in Go (with net/http) and supports Content-Type and Content-Encoding negotiation. It expects the optimized static assets to be contained under a root directory, as well as the index file (alt_path.json) that lists the relationships (e.g. alternate content type or content encoding) between variants of each asset. The server always returns to the client the smallest variant that the client supports, and supports revalidation/caching using the asset modification date. The appropriate Vary header is added to the response to ensure downstream caches can also correctly perform the content negotiation.

The server can optionally serve the static assets over HTTPS by providing the server image with a certificate and key in /server.crt and /server.key (it is recommended to mount these as secrets when the container is started, e.g. via Docker bind mounts or via Kubernetes secrets). When using HTTPS the server also supports HTTP/2.

The server image is based on gcr.io/distroless/static:nonroot: as such it contains no shell or other binaries apart from the standalone HTTP server above. The server does not need write access to the root filesystem of the container, so it is recommended to run with a read-only root filesystem (readOnlyRootFilesystem: true in Kubernetes). Similarly, the server does not need elevated privileges and runs with a non-root user, so it is recommended to disable running as root (runAsNonRoot: true in Kubernetes).

Usage

The simplest way to use this tool is the following:

1. Ensure you have Docker running

2. Place the static assets in the webroot directory

3. Run

   docker build . && \
   docker run -p 8080:80 $(docker image ls --format '{{.ID}}' | head -1)

   Please note that the second step is when asset optimization is performed and may take quite some time depending on how many static assets are present in webroot; if you want to speed up this step (at the expense of the compression ratio) you can replace docker build . with docker build --build-arg compression=LOW .

4. The static asset server should now be running on localhost:8080 (if you have file webroot/foo/bar.htm it should be served as localhost:8080/foo/bar.htm)

Builder

The builder image is automatically built and pushed to Docker Hub.

If you want to build it manually, e.g. to modify it and/or testing locally, run:

docker build --tag static_asset_builder --file Dockerfile.builder .

Examples

The assets directory contains two subdirectories: source contain random sample files in a variety of different formats, and optimized contains the optimized files and the variants that are then served by the static asset server.

The table below shows examples of how assets are optimized and served:

• The "Source" column links to the original asset
• The "Optimized variants" column links to the optimized assets as generated by the optimization pipeline
• The "Live demo" column points to an instance of the static asset server, serving the optimized asset: note that the Content-Type and Content-Encoding of the response is negotiated dynamically based on the Accept and Accept-Encoding headers in the request.

When testing the live demo, you can check in the developer console the negotiation result:

Source	Optimized variants	Live demo
a-chance-of-northern-lights.jpg (1479380 bytes)	a-chance-of-northern-lights.jpg (305494 bytes) a-chance-of-northern-lights.jpg.heif (148195 bytes) a-chance-of-northern-lights.jpg.jxl (227228 bytes) a-chance-of-northern-lights.jpg.webp (266378 bytes)	a-chance-of-northern-lights.jpg
bear.webp (132108 bytes)	bear.webp (132108 bytes)	bear.webp
example.json (3644 bytes)	example.json (2711 bytes) example.json.br (880 bytes) example.json.gz (990 bytes) example.json.zst (1012 bytes)	example.json
gradient.png (3974 bytes)	gradient.png (2034 bytes) gradient.png.avif (1470 bytes)	gradient.png
hourglass.gif (875 bytes)	hourglass.gif (746 bytes)	hourglass.gif
jquery-1.11.3.js (284394 bytes)	jquery-1.11.3.js (94991 bytes) jquery-1.11.3.js.br (29831 bytes) jquery-1.11.3.js.gz (31911 bytes) jquery-1.11.3.js.zst (31331 bytes)	jquery-1.11.3.js
kiss.gif (384825 bytes)	kiss.gif (371623 bytes) kiss.gif.webp (197796 bytes)	kiss.gif
make-it-new.jpg (1084975 bytes)	make-it-new.jpg (369427 bytes) make-it-new.jpg.avif (341575 bytes) make-it-new.jpg.heif (157154 bytes) make-it-new.jpg.jxl (334288 bytes) make-it-new.jpg.webp (338696 bytes)	make-it-new.jpg
pattern-bw.svg (75079 bytes)	pattern-bw.svg (65586 bytes) pattern-bw.svg.br (20391 bytes) pattern-bw.svg.gz (25009 bytes) pattern-bw.svg.zst (24241 bytes)	pattern-bw.svg
pattern-color.png (326860 bytes)	pattern-color.png (279412 bytes) pattern-color.png.avif (224374 bytes) pattern-color.png.heif (76768 bytes) pattern-color.png.jxl (125909 bytes) pattern-color.png.webp (162706 bytes)	pattern-color.png
reddit.html (982765 bytes)	reddit.html (982765 bytes) reddit.html.br (180965 bytes) reddit.html.gz (291533 bytes) reddit.html.zst (192426 bytes)	reddit.html
rose.webp (81978 bytes)	rose.webp (81978 bytes) rose.webp.avif (16494 bytes)	rose.webp
social.png (93404 bytes)	social.png (68363 bytes) social.png.avif (36122 bytes) social.png.heif (31536 bytes) social.png.jxl (49777 bytes) social.png.webp (41678 bytes)	social.png
sound-wave.svg (114176 bytes)	sound-wave.svg (78778 bytes) sound-wave.svg.br (881 bytes) sound-wave.svg.gz (1144 bytes) sound-wave.svg.zst (884 bytes)	sound-wave.svg
terminated.gif (63849 bytes)	terminated.gif (63849 bytes) terminated.gif.apng (56056 bytes) terminated.gif.webp (34418 bytes)	terminated.gif
wikipedia.html (81182 bytes)	wikipedia.html (81182 bytes) wikipedia.html.br (15326 bytes) wikipedia.html.gz (18541 bytes) wikipedia.html.zst (18084 bytes)	wikipedia.html

Contributing

PRs are welcome. Some ideas for what to add:

• General
  • Write tests
• Image formats
  • Add WebP optimization
  • Add AVIF optimization
  • Add all low efficiency variants for all formats, to improve compatibility (e.g. JPEG variant for AVIF files)
  • Add AVIF variant for WebP and GIF assets
  • Add HEIF (image/heif) variants for image assets
  • Add JPEG-XL (image/jxl) variants for image assets
  • Add JPEG-XL jxl content-encoding variant
  • Add WebP2 variants for image assets
  • Add BGP variants for image assets
• Other data formats
  • Add HTML minification
  • Add CSS minification
  • Dictionary compression
    • Add zstd/gzip/brotli dictionary generation
    • Add specialized dictionary generation (e.g. different dictionaries for different mimetypes)
    • Add dictionary negotiation
    • Add dictionary serving
  • Add LZMA content-encoding variants
• Optimization pipeline
  • Support caching optimization results
  • Use unique (guaranteed collision-free) file names for asset variants
  • Allow to control optimization on a per-file basis
  • Allow to disable optimization of certain formats (e.g. GIF files)
  • Allow to disable creation of certain variants (e.g. HEIF variants)
  • Automatic generation of lower resoluation variants (e.g. 1x/1.5x from 2x or from CSS-like selectors like max-width: 640px)
  • Automatic generation of lower quality variants (e.g. q=65, q=85, and lossless)
  • Automatic generation of lower decompression overhead variants (e.g. enable decoding with limited amounts of memory, or slow CPU)
  • Detect image type (e.g. graphics/drawing vs. photo) and source compression quality to decide which variants to generate (e.g. skip PNG for photos) and the variant compression parameters (e.g. quality, subsampling, filtering, ...)
• Asset server
  • Optionally embed assets in the server binary (go:embed)
  • Provide an optional way to serve variants based on a "first contentful paint" criteria (important for image formats that support progressive decoding)
  • Add ETag support
  • Allow to request a specifc content-type or content-encoding via query parameters (e.g. for use with source)
  • Allow to request the original/unoptimized asset
  • Decide whether to add Content-Location support
  • Add support for client hints (e.g. Save-Data, Device-Memory, Width, ...)
  • Decide whether to migrate to https://github.com/kevinpollet/nego
  • Use the caniuse.com database to augment content-type support (e.g. when the UA sends */* or image/*)