ZSTD or XZ compression for server

[NexusXe]

Describe your idea

Configurable ZSTD and/or XZ compression options would be very useful for this mod, as the inefficiency and CPU usage for creating very large ZIP backups can cause significant server lag and massive world backup files.

Using xz on Linux creates a world backup anywhere from ~35% to ~65% smaller than zip in an equivalent time, and for server admins who frequently create backups (like me!) the reduced disk I/O and compression time would come in handy.

[offbeat-stuff]

a lot more can saved by using diffs (like github)
only saving what is changed in files

[MelanX]

I started looking into adding another compression method. However, only saving diffs is already possible. (But this feature needs a bit more improvement I guess)

[MelanX]

I looked a bit deeper into it and had a good system to compress using ZSTD and XZ. However, this was only theoretically and not able to compress in game since it couldn't find the correct classes. As reference how I tried implementing it, I leave this Gist here: https://gist.github.com/MelanX/6b3924bbcf7157f4a200d8309f89dffd
Maybe someone else can use it and add it in a pull request.

More work done:
https://github.com/MelanX/SimpleBackups/tree/feature/xz-backup

[Angelin01]

@MelanX Sorry for necroing this old issue, but I think if your problems were finding classes, you might just have needed to include dependencies in the final jar. For example, I know Distant Horizons has both LZMA (xz) and LZ4 compression. They use the shadow Gradle plugin to shadow the dependencies, which can be found here: https://github.com/johnrengelman/shadow

With that, you should be able to properly include the dependencies in the mod. HOWEVER...

I know shadowing can cause some issues if multiple copies of the same jar are included, for example, if you also decided to add org.tukaani:xz in a different version than Distant Horizons, so maybe there's a better way to go about this.

I'm also interested in a slightly better compression than ZIP (albeit XZ tends to be CPU intensive), so I'm willing to help if I can (however I have zero knowledge specific to Minecraft modding).

This is a little outdated, but might help: https://gist.github.com/SizableShrimp/949e7c219bfc94487a45226b64ac7749. It includes a "relocate" step that seems to avoid the issue I mentioned above.

[MelanX]

Thank you very much for trying to help :)
I really want to implement that one, too!
As you can see here, I already added it as dependency using the forge intern feature Jar-in-Jar: https://github.com/ChaoticTrials/SimpleBackups/blob/feature/xz-backup/build.gradle#L11
I’m not sure anymore what the exact problem for me was now, but I think it was that the file size was nearly identical to zip and it took even longer. Here my way to implement it: https://github.com/ChaoticTrials/SimpleBackups/blob/feature/xz-backup/src/main/java/de/melanx/simplebackups/compression

If I’m doing this wrong and you (or anyone else) can help with this, I’d really love it!

[Angelin01]

I'm actually trying to set it up right now! I was willing to test Zstd, since it has a native Java implementation too, and could be a better middle ground than XZ and the others.

However, ModUtils is kicking my ass while trying to just add the Shadow plugin. This guide gets me an error:

> Failed to apply plugin 'org.moddingx.modgradle.mapping'.
   > Cannot add extension with name 'modgradle', as there is an extension already registered with that name.

No matter if I apply the plugin as

buildscript {
    apply from: 'https://moddingx.github.io/ModUtils/v5/buildscript.gradle', to: buildscript
    repositories {
        gradlePluginPortal()
    }
    dependencies {
        classpath 'com.github.johnrengelman:shadow:8.1.1'
    }
}

apply from: 'https://moddingx.github.io/ModUtils/v5/mod.gradle'
apply plugin: 'org.moddingx.modgradle.cursedep'
apply plugin: 'maven-publish'
apply plugin: 'com.github.johnrengelman.shadow'

Or

buildscript {
    apply from: 'https://moddingx.github.io/ModUtils/v5/buildscript.gradle', to: buildscript
}

plugins {
    id 'com.github.johnrengelman.shadow' version '8.1.1'
}

apply from: 'https://moddingx.github.io/ModUtils/v5/mod.gradle'
apply plugin: 'org.moddingx.modgradle.cursedep'
apply plugin: 'maven-publish'

I'm not used to Gradle, but my three attempts are playing with mods have left me with a giant hate for it.

[Angelin01]

Very interesting. I decided to do some basic tests, since I wasn't getting the plugin to work and you said that XZ was very similar to zip.

I used 7zips LZMA2 implementation, just because that's what I had installed. The results are quite shocking:

Format	Size in bytes
Raw	161.501.184
7z Ultra	113.864.704
7z Normal	114.577.408
Zip Normal	118.218.752
Zip Ultra	117.891.072

I'm actually shocked at how poorly LZMA2 did. I'm thinking Minecraft's data just doesn't compress well, and whatever gains are mostly from archiving the files together and thus creating some "repeating information".

I'll... Give up on this for now. It simply looks like it's not worth the trouble for such small gains. Thank you for the great mod!

PS: LZ4 might still be worth it. It's super fast, and if the loss of ratio isn't bad, could be a speedy alternative to zip.

[Git-North]

Sorry for the second necro of this post but I've got some interesting resources. A small disclaimer, I don't understand java modding, only compression algorithms. I got 2 that may be interesting.

One of them is FLZMA2 (f stands for fast) it's compression ratio is very slightly bigger then 7z but it does increase the speed by quite a lot thanks to a few optimizations (almost double the speed), it also has backwards compatibility with (non f)lzma2 supporting archivers

Other one is zpaqfranz I've personally seen it compress "uncompressible" data to x0.8 size, it also supports incremantal backups. It takes a lot of time to compress relatively large data though.

as for ZTSD, it's fast, it's got good ratio's between speed and compression and unlike these 2 i would assume it's a lot more available to implement. I use it on my linux filesystem's default live compressor and could be very well used as a drop-in replacement for zlib in terms of compression speed while achieving greater compression

[wolfieboy09]

A little late to this, but I found this git that's last updated 2 years ago: https://github.com/IceSelkie/MinecraftWorldCompressor

[offbeat-stuff]

Just use btrfs filesystem

[NexusXe]

Just use btrfs filesystem

Using a filesystem with compression wouldn't really help here, as the kind of compression we're doing for backups hugely benefits from solid compression provided by wrappers like .xz or .zstd (or .zip, for that matter) instead of block-level filesystem compression ZFS, BTRFS, or BCachefs provide.

[HoldYourWaffle]

Filesystem-level compression doesn't help with cloud storage usage either.

[Git-North]

Just use btrfs filesystem

Zpaq can use diffs