Skip to content

Commit 21d5c60

Browse files
Add custom schema serialization format (#23)
Resolves #9 Replaced schema encoding on the wire by a simple binary format instead of JSON. The format has not much extensibility built-in but any change to the schema descriptor format is going to be a breaking change anyway, so this simple approach that is fast and compact works fine for our needs. For Otel/STEF example schema we get the following byte sizes: ``` Uncompressed Zstd JSON 1877 498 Binary 579 308 Diff: -69% -38% ``` Fuzzing discovered a few deserialization bugs, which I fixed and then was run without failures for another hour. Alternates Considered: Msgpack, CBOR, Protobuf. All of these would be extra dependencies that are not necessarily available as a core library in every language, which would complicate STEF implementation. This simple binary format is trivial to implement in any language and avoids bringing a fairly complicated dependency.
1 parent 46aac76 commit 21d5c60

File tree

10 files changed

+1022
-24
lines changed

10 files changed

+1022
-24
lines changed

go/otel/oteltef/metricswriter.go

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

go/otel/oteltef/spanswriter.go

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

go/pkg/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ module github.com/splunk/stef/go/pkg
33
go 1.23.2
44

55
require (
6+
github.com/google/go-cmp v0.6.0
67
github.com/klauspost/compress v1.17.8
78
github.com/stretchr/testify v1.9.0
89
)

go/pkg/go.sum

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
22
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
3+
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
4+
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
35
github.com/klauspost/compress v1.17.8 h1:YcnTYrq7MikUT7k0Yb5eceMmALQPYBW/Xltxn0NAMnU=
46
github.com/klauspost/compress v1.17.8/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
57
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=

0 commit comments

Comments
 (0)