Add custom schema serialization format

Resolves #9 Replaced schema encoding on the wire by a simple binary format instead of JSON. The format has not much extensibility built-in but any change to the schema descriptor format is going to be a breaking change anyway, so this simple approach that is fast and compact works fine for our needs. For Otel/STEF example schema we get the following byte sizes: ``` Uncompressed Zstd JSON 1877 498 Binary 579 308 Diff: -69% -38% ``` Alternates Considered: Msgpack, CBOR, Protobuf. All of these would be extra dependencies that are not necessarily available as a core library in every language, which would complicate STEF implementation. This simple binary format is trivial to implement in any language and avoids bringing a fairly complicated dependency.
splunk · Jan 25, 2025 · 3f4a68a · 3f4a68a
1 parent 46aac76
commit 3f4a68a
Show file tree

Hide file tree

Showing 10 changed files with 961 additions and 24 deletions.
diff --git a/go/otel/oteltef/metricswriter.go b/go/otel/oteltef/metricswriter.go
diff --git a/go/otel/oteltef/spanswriter.go b/go/otel/oteltef/spanswriter.go
diff --git a/go/pkg/go.mod b/go/pkg/go.mod
@@ -3,6 +3,7 @@ module github.com/splunk/stef/go/pkg
 go 1.23.2
 
 require (
+	github.com/google/go-cmp v0.6.0
 	github.com/klauspost/compress v1.17.8
 	github.com/stretchr/testify v1.9.0
 )

diff --git a/go/pkg/go.sum b/go/pkg/go.sum
@@ -1,5 +1,7 @@
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
+github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/klauspost/compress v1.17.8 h1:YcnTYrq7MikUT7k0Yb5eceMmALQPYBW/Xltxn0NAMnU=
 github.com/klauspost/compress v1.17.8/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=