compression experiments

[marcustyphoon]

Currently our unshortened URLs are compressed via LZMA (json-url library), which is unusably slow if we store all of the results. Our short URLs, before being sent to the server to be shortened, are compressed using deflate (pako), which has a bit worse than half its compression ratio if we do so.

Experimenting here with some brotli/zstd libraries. Brotli 8 and zstd 8 seem like reasonable levels, yielding similar compression ratios as json-url LZMA and similar performance to pako's deflate.

zstd is faster in this particular setup, but seems to break when reloaded by vite's hot module reloading in dev mode.

Important considerations include bundle size (brotli usually ships a large dictionary with its compressor; I think this includes it; would like to see the results without it) and future compatibility ("Content-Encoding", "br" exists; node:zlib supports brotli and cloudflare workers ships it with nodejs_compat_v2 if we ever needed to decode on the server side; none of these are currently true of zstd).

jsonString.length 2341796
json-url length 34586
Compressed json-url data in:: 15717.1650390625 ms
pakoData length 97403
Compressed pakoData data in:: 51.894775390625 ms
brotliData 0 length 1147510
Compressed brotli 0 data in:: 167.887939453125 ms
brotliData 1 length 1118283
Compressed brotli 1 data in:: 159.541015625 ms
brotliData 2 length 66170
Compressed brotli 2 data in:: 79.85107421875 ms
brotliData 3 length 57203
Compressed brotli 3 data in:: 82.751220703125 ms
brotliData 4 length 51322
Compressed brotli 4 data in:: 96.630126953125 ms
brotliData 5 length 42127
Compressed brotli 5 data in:: 57.64013671875 ms
brotliData 6 length 41082
Compressed brotli 6 data in:: 69.693115234375 ms
brotliData 7 length 40336
Compressed brotli 7 data in:: 80.15087890625 ms
brotliData 8 length 39956
Compressed brotli 8 data in:: 85.27099609375 ms
brotliData 9 length 39471
Compressed brotli 9 data in:: 106.369140625 ms
brotliData 10 length 36894
Compressed brotli 10 data in:: 533.053955078125 ms
brotliData 11 length 35382
Compressed brotli 11 data in:: 2325.001953125 ms
zstd 0 length 55255
Compressed zstd 0 data in:: 7.204833984375 ms
zstd 1 length 67679
Compressed zstd 1 data in:: 7.087158203125 ms
zstd 2 length 57788
Compressed zstd 2 data in:: 5.81884765625 ms
zstd 3 length 55255
Compressed zstd 3 data in:: 6.541015625 ms
zstd 4 length 54687
Compressed zstd 4 data in:: 7.327880859375 ms
zstd 5 length 47556
Compressed zstd 5 data in:: 15.64501953125 ms
zstd 6 length 44647
Compressed zstd 6 data in:: 20.198974609375 ms
zstd 7 length 43686
Compressed zstd 7 data in:: 23.38818359375 ms
zstd 8 length 42258
Compressed zstd 8 data in:: 21.94384765625 ms
zstd 9 length 41852
Compressed zstd 9 data in:: 28.1669921875 ms
zstd 10 length 41287
Compressed zstd 10 data in:: 39.4228515625 ms
zstd 11 length 40823
Compressed zstd 11 data in:: 42.204833984375 ms
zstd 12 length 40822
Compressed zstd 12 data in:: 57.587890625 ms
zstd 13 length 41243
Compressed zstd 13 data in:: 115.190185546875 ms
zstd 14 length 40599
Compressed zstd 14 data in:: 183.5478515625 ms
zstd 15 length 40108
Compressed zstd 15 data in:: 276.432861328125 ms
zstd 16 length 38955
Compressed zstd 16 data in:: 179.58203125 ms
zstd 17 length 38562
Compressed zstd 17 data in:: 208.568115234375 ms
zstd 18 length 38892
Compressed zstd 18 data in:: 252.544189453125 ms
zstd 19 length 37714
Compressed zstd 19 data in:: 516.73388671875 ms
zstd 20 length 37714
Compressed zstd 20 data in:: 524.890869140625 ms
zstd 21 length 37490
Compressed zstd 21 data in:: 1162.782958984375 ms

[draft previews]

[github-actions]

Deployed preview build to Cloudflare!

Last commit:	6045119
Preview URL:	https://7ea47555.discretize-gear-optimizer.pages.dev
Branch Preview URL:	https://compression-experiments.discretize-gear-optimizer.pages.dev

[marcustyphoon]

Interestingly, zlib.brotliCompressSync(data, { params: { [zlib.constants.BROTLI_PARAM_QUALITY]: 5 }, }); in the cloudflare function in local testing took 7ms, which is an order of magnitude faster than the webassembly brotli this runs on the client. Unfortunately, we have a 10ms cpu time limit for function invocations on the cloudflare free tier, which I don't want to run into, so I'd rather spend the time on the client.