Add JDA integration module

README.md

@@ -12,54 +12,35 @@ Lightweight modular support for Zstandard streaming decompression, for JVM Disco
 - Windows: x86-64, aarch64
 - macOS (darwin): x86-64, aarch64
 
-## 🤖 For bot developers
+## 🔥 For JDA users
 
-### Installation
+See the [JDA integration module](jda-integration).
 
-You're likely here if you want to use Zstd decompression for your Discord bot!
-
-[![discord-zstd-java-jni-impl on Maven Central][jni-impl-maven-central-shield] ][jni-impl-maven-central-link]
-
-This is compatible with Java 8+.
-
-#### Gradle
-```kotlin
-dependencies {
-    runtimeOnly("dev.freya02:discord-zstd-java-jni-impl:VERSION") // TODO replace VERSION with current release
-}
-```
-
-#### Maven
-```xml
-<dependency>
-    <groupId>dev.freya02</groupId>
-    <artifactId>discord-zstd-java-jni-impl</artifactId>
-    <version>VERSION</version> <!-- TODO replace VERSION with current release -->
-    <scope>runtime</scope>
-</dependency>
-```
-
-> [!TIP]
-> To remove the warning when the natives are loaded, add `--enable-native-access=ALL-UNNAMED` to your JVM arguments.
+## 📖 For library developers
 
-### Usage
-As a bot developer, you don't need to do anything.
+[![discord-zstd-java-api on Maven Central][api-maven-central-shield] ][api-maven-central-link]
 
-If you want to load a different version of the native library,
-you can do so by calling `ZstdNativesLoader.load(Path)` or `loadFromJar(String)`. These functions will return `false` if the natives were already loaded, as they can't be replaced.
+### Built-in integration
 
-## 📖 For library developers
-### Installation
+If you decide to integrate this library into yours,
+you will only need the `dev.freya02:discord-zstd-java-api:VERSION` dependency, it is compatible with Java 8+.
 
-[![discord-zstd-java-api on Maven Central][api-maven-central-shield] ][api-maven-central-link]
+Your users will need to install an implementation, we recommend using `discord-zstd-java-jni-impl`.
 
-You will only need the `dev.freya02:discord-zstd-java-api:VERSION` dependency, it is compatible with Java 8+.
+The users can also load a different version of the native library,
+they can do so by calling `ZstdNativesLoader.load(Path)` or `loadFromJar(String)`.
+These functions will return `false` if the natives were already loaded, as they can't be replaced.
 
-### Usage
+#### Usage
 
 The main interface is `DiscordZstd`, you can get an instance with `DiscordZstdProvider.get()`.
 Then, you can either:
 1. Do bulk processing with a decompressor obtained with `DiscordZstd#createDecompressor` and kept per gateway connection,
    calling `ZstdDecompressor#decompress` on each gateway message
 2. Process gradually with a context obtained from `DiscordZstd#createContext` and kept per gateway connection,
-    then making input streams with `ZstdContext#createInputStream` from each gateway message
+   then making input streams with `ZstdContext#createInputStream` from each gateway message
+
+### External integration
+
+You can also provide an API in your library, this way users can choose to use any decompression library they want.
+After creating your API, please submit a pull request here with a new module which implements that API.

gradle/libs.versions.toml

@@ -1,13 +1,16 @@
 [libraries]
+assertj = "org.assertj:assertj-core:3.27.6"
 jackson-databind = "tools.jackson.core:jackson-databind:3.0.3"
 jda = "net.dv8tion:JDA:6.1.0"
+jda-snapshot = "net.dv8tion:JDA:6.2.0_DEV"
 jna = "net.java.dev.jna:jna:5.18.1"
 jreleaser = "org.jreleaser:org.jreleaser.gradle.plugin:1.20.0"
 jspecify = "org.jspecify:jspecify:1.0.0"
 junit = "org.junit.jupiter:junit-jupiter:5.14.0"
 junit-launcher = "org.junit.platform:junit-platform-launcher:1.14.0"
 kotlin-logging-jvm = "io.github.oshai:kotlin-logging-jvm:7.0.13"
 kotlinx-coroutines-core = "org.jetbrains.kotlinx:kotlinx-coroutines-core:1.10.2"
+mockito = "org.mockito:mockito-core:5.20.0"
 slf4j = "org.slf4j:slf4j-api:2.0.17"
 logback-classic = "ch.qos.logback:logback-classic:1.5.21"
 trove4j-core = "net.sf.trove4j:core:3.1.0"

jda-integration/BENCHMARKS.md

@@ -0,0 +1,44 @@
+# Benchmarks - Compared to JDA's Zlib decompression
+
+## Overview
+
+Zstd decompression is beneficial for larger bots where the CPU and GC allocations improvements will be the most noticeable, and for Discord itself. Smaller bots will only see slight benefits.
+
+In both cases, payloads are slightly smaller mostly during startup, and quite similar during normal operation.
+
+In the best case, when decompressing and deserializing, we see a 31% speed improvement with half the memory allocations.
+
+## Synthetic benchmarks
+
+> [!NOTE]
+> Thanks to @MrPowerGamerBR for helping me gather some data from their bot and running benchmarks on their machine!
+
+> [!TIP]
+> "bulk" here means going from compressed `byte[]` to a fully decompressed `byte[]`.
+
+With 10 shards starting up from scratch, giving us 364 MB of decompressed data, they ran [a few benchmarks](../benchmarks/src/jmh/java/dev/freya02/discord/zstd) comparing zlib, zstd, as well as their bulk (what JDA currently does) and stream (letting Jackson consume an `InputStream`) variants.
+
+They were run on their production server, with an AMD Ryzen 5 5600X and produced [this data](https://gist.github.com/freya022/0516a809d43ee1d084ed205ac4fbe56c).
+
+If we look at the complete package, transforming the compressed data into a usable `DataObject`, we can see bulk-decompressing with Zstd cut time spent by 30% and reduced GC allocations by 35%
+
+However, if we use streaming, Zlib gets about 10% of speed increase and removes 48% of GC allocations, while Zstd gets a 31% speed improvement and the same memory improvements.
+
+### Other benchmarks
+
+I have also run some numbers on my machine (an AMD Ryzen 7 3700X, with boost disabled), not much different here:
+
+- 10 shards of [randomly generated](../test-data-generator) data: https://gist.github.com/freya022/04d5e8cf7d44c9680ae42154808cddfd
+- A bot with a single guild: https://gist.github.com/freya022/8922140965bc51a699b135ebc2f96914
+
+## Runtime statistics
+
+They were also kind enough to run their ~2.5K shards for a day while recording the compressed size, decompressed size and time-to-decompress.
+
+The shards were split in 3 equally-sized sets:
+
+- Using the current Zlib implementation
+- Using Zstd with a 8 KB buffer
+- Using Zstd with a 128 KB buffer
+
+The results can be seen there: https://gist.github.com/freya022/7b35aa412a4f125ca1b139b71360ab45

jda-integration/README.md

@@ -0,0 +1,63 @@
+[api-maven-central-shield]: https://img.shields.io/maven-central/v/dev.freya02/discord-zstd-java-api?label=Maven%20central&logo=apachemaven
+[api-maven-central-link]: https://central.sonatype.com/artifact/dev.freya02/discord-zstd-java-api
+[jda-integration-maven-central-shield]: https://img.shields.io/maven-central/v/dev.freya02/discord-zstd-java-jda-integration?label=Maven%20central&logo=apachemaven
+[jda-integration-maven-central-link]: https://central.sonatype.com/artifact/dev.freya02/discord-zstd-java-jda-integration
+
+# discord-zstd-java - JDA integration
+
+Lightweight Zstandard decompression for the Java Discord API. (JDA)
+
+This module is typically useful for sharded (large) bots, if you are interested in the performance difference, see [here](BENCHMARKS.md).
+
+## Installation
+
+[![discord-zstd-java-jda-integration on Maven Central][jda-integration-maven-central-shield] ][jda-integration-maven-central-link]
+
+This is compatible with Java 8+.
+
+### Gradle
+```kotlin
+dependencies {
+    implementation("dev.freya02:discord-zstd-java-jda-integration:VERSION") // TODO replace VERSION with current release
+}
+```
+
+### Maven
+```xml
+<dependency>
+    <groupId>dev.freya02</groupId>
+    <artifactId>discord-zstd-java-jda-integration</artifactId>
+    <version>VERSION</version> <!-- TODO replace VERSION with current release -->
+</dependency>
+```
+
+> [!TIP]
+> To remove the warning when the natives are loaded, add `--enable-native-access=ALL-UNNAMED` to your JVM arguments.
+
+## Usage
+
+JDA's gateway decompressor can be configured in `GatewayConfig.Builder`, in two ways:
+
+- `useBufferedTransportDecompression` lets you use decompress payloads all at once, this is what JDA does by default.
+- `useStreamedTransportDecompression` lets you decompress payloads progressively, this means no memory allocations for decompression, but also prevents from printing corrupted payloads (though extremely rare)
+
+If you choose to use buffered decompression, use `ZstdBufferedTransportGatewayDecompressor`, if you want to use streamed decompression, use `ZstdStreamedTransportGatewayDecompressor`.
+
+For example:
+
+```java
+void main(String[] args) {
+    DefaultShardManagerBuilder
+            .createDefault(args[0])
+            .setGatewayConfig(
+                    GatewayConfig.builder()
+                            .useStreamedTransportDecompression(ZstdStreamedTransportGatewayDecompressor.supplier())
+                            .build())
+            .build();
+}
+```
+
+## Overriding natives
+
+If you want to load a different version of the native library,
+you can do so by calling `ZstdNativesLoader.load(Path)` or `loadFromJar(String)`. These functions will return `false` if the natives were already loaded, as they can't be replaced.

jda-integration/build.gradle.kts

@@ -0,0 +1,68 @@
+plugins {
+    `java-conventions`
+    `java-library`
+    `publish-conventions`
+}
+
+val fullProjectName = "${rootProject.name}-${project.name}"
+
+tasks.withType<Jar> {
+    archiveBaseName = fullProjectName
+}
+
+repositories {
+    mavenLocal()
+}
+
+val mockitoAgent by configurations.creating
+dependencies {
+    api(project(":api"))
+    runtimeOnly(project(":jni-impl"))
+
+    implementation(libs.jda.snapshot)
+
+    //Code safety
+    compileOnly(libs.jspecify)
+    testCompileOnly(libs.jspecify)
+
+    //Logger
+    implementation(libs.slf4j)
+
+    // JUnit 5 (JUnit 6 is not Java 8 compatible)
+    testImplementation(libs.bundles.junit)
+    testImplementation(libs.mockito)
+    mockitoAgent(libs.mockito) { isTransitive = false }
+    testImplementation(libs.assertj)
+
+    testRuntimeOnly(libs.logback.classic)
+}
+
+java {
+    withJavadocJar()
+    withSourcesJar()
+}
+
+tasks.named<JavaCompile>("compileJava") {
+    options.release.set(8)
+}
+
+tasks.test {
+    useJUnitPlatform()
+    failFast = false
+
+    jvmArgs("-javaagent:${mockitoAgent.asPath}")
+}
+
+tasks.jar {
+    manifest {
+        attributes("Automatic-Module-Name" to "discord.zstd.java.jda.integration")
+    }
+}
+
+registerPublication(
+    name = fullProjectName,
+    description = "Lightweight Zstandard decompressor for JDA",
+    url = "https://github.com/freya022/discord-zstd-java/tree/master/jda-integration",
+) {
+    from(components["java"])
+}

jda-integration/src/main/java/dev/freya02/discord/zstd/jda/ZstdBufferedTransportGatewayDecompressor.java

@@ -0,0 +1,87 @@
+package dev.freya02.discord.zstd.jda;
+
+import dev.freya02.discord.zstd.api.*;
+import net.dv8tion.jda.api.exceptions.DecompressionException;
+import net.dv8tion.jda.api.requests.gateway.compression.GatewayDecompressor;
+import org.jspecify.annotations.NullMarked;
+import org.jspecify.annotations.Nullable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.function.Supplier;
+
+/**
+ * Provides buffered transport-level decompression of gateway messages for the Java Discord API (JDA).
+ *
+ * @see #supplier(int)
+ */
+@NullMarked
+public class ZstdBufferedTransportGatewayDecompressor implements GatewayDecompressor.Transport.Buffered {
+    private static final Logger LOGGER = LoggerFactory.getLogger(ZstdBufferedTransportGatewayDecompressor.class);
+
+    private final DiscordZstdDecompressor decompressor;
+
+    public ZstdBufferedTransportGatewayDecompressor(DiscordZstdDecompressor decompressor) {
+        this.decompressor = decompressor;
+    }
+
+    /**
+     * Creates a supplier of {@link ZstdBufferedTransportGatewayDecompressor} with the provided decompression buffer size.
+     *
+     * <h4>Buffer sizes</h4>
+     * This defines the size, in bytes, of the intermediate buffer used for decompression,
+     * larger buffer means less decompression loops at a fixed cost of memory.
+     *
+     * <ul>
+     *     <li>The recommended value is {@value DiscordZstdDecompressor#DEFAULT_BUFFER_SIZE}, as it is sufficient for most Discord payloads</li>
+     *     <li>
+     *         A value "recommended" by Zstd is set with {@link DiscordZstdDecompressor#ZSTD_RECOMMENDED_BUFFER_SIZE};
+     *         However it is not recommended for normal use cases, see the docs for more details.
+     *     </li>
+     *     <li>The minimum is {@value DiscordZstdDecompressor#MIN_BUFFER_SIZE}</li>
+     * </ul>
+     *
+     * @param  bufferSizeHint
+     *         The hint or value for the size of the buffer used for decompression
+     *
+     * @throws IllegalArgumentException
+     *         If {@code bufferSize} is less than {@value DiscordZstdDecompressor#MIN_BUFFER_SIZE} and not {@value DiscordZstdDecompressor#ZSTD_RECOMMENDED_BUFFER_SIZE}
+     *
+     * @return A new supplier of {@link ZstdBufferedTransportGatewayDecompressor}
+     */
+    public static Supplier<ZstdBufferedTransportGatewayDecompressor> supplier(int bufferSizeHint) {
+        DiscordZstd zstd = DiscordZstdProvider.get();
+        DiscordZstdDecompressorFactory factory = zstd.createDecompressorFactory(bufferSizeHint);
+        return () -> new ZstdBufferedTransportGatewayDecompressor(factory.create());
+    }
+
+    @Nullable
+    @Override
+    public String getQueryParameter() {
+        return "zstd-stream";
+    }
+
+    @Override
+    public void reset() {
+        decompressor.reset();
+    }
+
+    @Override
+    public void shutdown() {
+        decompressor.close();
+    }
+
+    @Override
+    public byte[] decompress(byte[] data) throws DecompressionException {
+        if (LOGGER.isTraceEnabled()) {
+            LOGGER.trace("Decompressing data {}", Arrays.toString(data));
+        }
+
+        try {
+            return decompressor.decompress(data);
+        } catch (DiscordZstdException e) {
+            throw new DecompressionException(e);
+        }
+    }
+}