Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vtprotobuf: Reuse backing array with referenced structs instead of pooling individual objects #63

Merged

Conversation

carsonip
Copy link
Member

@carsonip carsonip commented Aug 1, 2023

Alternative to #37 , but using vtprotobuf fork that reuses backing array including referenced structs, which has an upstream PR. With this, we will no longer be pooling Keyed* individually. Either with this or #37, there will be significantly fewer garbage.

Closes #20

benchstat:

goos: linux
goarch: amd64
pkg: github.com/elastic/apm-aggregation/aggregators
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
                                       │ bench-out/vt-unmarshal-pooling-including-structs-2-before │ bench-out/vt-unmarshal-pooling-including-structs-2-after │
                                       │                          sec/op                           │              sec/op                vs base               │
AggregateCombinedMetrics-16                                                            2.001µ ± 1%                         1.926µ ± 1%   -3.75% (p=0.002 n=6)
AggregateBatchSerial-16                                                                5.227µ ± 1%                         4.489µ ± 5%  -14.11% (p=0.002 n=6)
AggregateBatchParallel-16                                                              5.524µ ± 9%                         5.008µ ± 3%   -9.33% (p=0.002 n=6)
CombinedMetricsEncoding-16                                                            1410.0n ± 3%                         878.5n ± 2%  -37.69% (p=0.002 n=6)
CombinedMetricsToBatch-16                                                              61.90µ ± 1%                         61.41µ ± 2%        ~ (p=0.132 n=6)
EventToCombinedMetrics-16                                                              420.7n ± 2%                         418.4n ± 3%        ~ (p=0.240 n=6)
NDJSONSerial/go-2.0.0.ndjson-16                                                        19.11m ± 3%                         16.84m ± 4%  -11.88% (p=0.002 n=6)
NDJSONSerial/nodejs-3.29.0.ndjson-16                                                  10.980m ± 4%                         9.488m ± 3%  -13.59% (p=0.002 n=6)
NDJSONSerial/python-6.7.2.ndjson-16                                                    27.13m ± 4%                         23.21m ± 3%  -14.45% (p=0.002 n=6)
NDJSONSerial/ruby-4.5.0.ndjson-16                                                      16.14m ± 1%                         13.95m ± 4%  -13.57% (p=0.002 n=6)
NDJSONParallel/go-2.0.0.ndjson-16                                                      19.17m ± 4%                         16.71m ± 7%  -12.85% (p=0.002 n=6)
NDJSONParallel/nodejs-3.29.0.ndjson-16                                                11.005m ± 3%                         9.518m ± 6%  -13.51% (p=0.002 n=6)
NDJSONParallel/python-6.7.2.ndjson-16                                                  27.26m ± 2%                         23.03m ± 3%  -15.54% (p=0.002 n=6)
NDJSONParallel/ruby-4.5.0.ndjson-16                                                    16.25m ± 3%                         13.98m ± 3%  -13.97% (p=0.002 n=6)
geomean                                                                                458.6µ                              399.0µ       -13.01%

                                       │ bench-out/vt-unmarshal-pooling-including-structs-2-before │ bench-out/vt-unmarshal-pooling-including-structs-2-after │
                                       │                           B/op                            │             B/op              vs base                    │
AggregateCombinedMetrics-16                                                        1.416Ki ±  0%                     1.104Ki ± 1%   -22.01% (p=0.002 n=6)
AggregateBatchSerial-16                                                            2.459Ki ±  2%                     1.373Ki ± 3%   -44.17% (p=0.002 n=6)
AggregateBatchParallel-16                                                          2.500Ki ±  3%                     1.370Ki ± 3%   -45.21% (p=0.002 n=6)
CombinedMetricsEncoding-16                                                           855.0 ± 44%                         0.0 ± 0%  -100.00% (p=0.002 n=6)
CombinedMetricsToBatch-16                                                          3.986Ki ±  3%                     3.982Ki ± 0%         ~ (p=0.502 n=6)
EventToCombinedMetrics-16                                                            0.000 ±  0%                       0.000 ± 0%         ~ (p=1.000 n=6) ¹
NDJSONSerial/go-2.0.0.ndjson-16                                                   11.591Mi ±  0%                     9.249Mi ± 1%   -20.21% (p=0.002 n=6)
NDJSONSerial/nodejs-3.29.0.ndjson-16                                               7.007Mi ±  0%                     5.589Mi ± 1%   -20.23% (p=0.002 n=6)
NDJSONSerial/python-6.7.2.ndjson-16                                                19.59Mi ±  1%                     15.32Mi ± 1%   -21.78% (p=0.002 n=6)
NDJSONSerial/ruby-4.5.0.ndjson-16                                                 10.418Mi ±  0%                     8.261Mi ± 0%   -20.71% (p=0.002 n=6)
NDJSONParallel/go-2.0.0.ndjson-16                                                 11.597Mi ±  0%                     9.233Mi ± 1%   -20.38% (p=0.002 n=6)
NDJSONParallel/nodejs-3.29.0.ndjson-16                                             7.011Mi ±  0%                     5.563Mi ± 1%   -20.66% (p=0.002 n=6)
NDJSONParallel/python-6.7.2.ndjson-16                                              19.63Mi ±  1%                     15.32Mi ± 1%   -21.94% (p=0.002 n=6)
NDJSONParallel/ruby-4.5.0.ndjson-16                                               10.424Mi ±  0%                     8.256Mi ± 0%   -20.80% (p=0.002 n=6)
geomean                                                                                          ²                                 ?                      ² ³
¹ all samples are equal
² summaries must be >0 to compute geomean
³ ratios must be >0 to compute geomean

                                       │ bench-out/vt-unmarshal-pooling-including-structs-2-before │ bench-out/vt-unmarshal-pooling-including-structs-2-after │
                                       │                         allocs/op                         │            allocs/op             vs base                 │
AggregateCombinedMetrics-16                                                           21.00 ± 0%                          17.00 ± 0%  -19.05% (p=0.002 n=6)
AggregateBatchSerial-16                                                               26.00 ± 4%                          13.00 ± 8%  -50.00% (p=0.002 n=6)
AggregateBatchParallel-16                                                             26.50 ± 6%                          13.00 ± 8%  -50.94% (p=0.002 n=6)
CombinedMetricsEncoding-16                                                            0.000 ± 0%                          0.000 ± 0%        ~ (p=1.000 n=6) ¹
CombinedMetricsToBatch-16                                                             95.00 ± 0%                          95.00 ± 0%        ~ (p=1.000 n=6) ¹
EventToCombinedMetrics-16                                                             0.000 ± 0%                          0.000 ± 0%        ~ (p=1.000 n=6) ¹
NDJSONSerial/go-2.0.0.ndjson-16                                                      142.1k ± 0%                         114.1k ± 2%  -19.66% (p=0.002 n=6)
NDJSONSerial/nodejs-3.29.0.ndjson-16                                                 79.18k ± 1%                         63.79k ± 1%  -19.43% (p=0.002 n=6)
NDJSONSerial/python-6.7.2.ndjson-16                                                  215.4k ± 1%                         162.7k ± 1%  -24.49% (p=0.002 n=6)
NDJSONSerial/ruby-4.5.0.ndjson-16                                                   124.53k ± 0%                         98.87k ± 0%  -20.61% (p=0.002 n=6)
NDJSONParallel/go-2.0.0.ndjson-16                                                    142.0k ± 0%                         113.9k ± 1%  -19.81% (p=0.002 n=6)
NDJSONParallel/nodejs-3.29.0.ndjson-16                                               79.20k ± 0%                         63.17k ± 1%  -20.24% (p=0.002 n=6)
NDJSONParallel/python-6.7.2.ndjson-16                                                215.4k ± 2%                         163.2k ± 1%  -24.23% (p=0.002 n=6)
NDJSONParallel/ruby-4.5.0.ndjson-16                                                 124.54k ± 1%                         99.09k ± 0%  -20.43% (p=0.002 n=6)
geomean                                                                                          ²                                    -22.21%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

macro benchmark:

                           │ github/run82.txt │         github/run83-vt.txt         │
                           │    events/sec    │  events/sec   vs base               │
10000AggregationGroups-512       13.76k ± 35%   15.90k ± 32%        ~ (p=0.132 n=6)
1000Transactions-512             13.72k ± 33%   18.83k ± 28%        ~ (p=0.065 n=6)
AgentAll-512                     9.124k ± 19%   8.998k ± 50%        ~ (p=0.818 n=6)
AgentGo-512                      11.15k ± 22%   12.01k ± 53%        ~ (p=0.310 n=6)
AgentNodeJS-512                  5.777k ± 28%   6.611k ± 29%        ~ (p=0.093 n=6)
AgentPython-512                  6.825k ± 26%   7.422k ± 76%        ~ (p=0.180 n=6)
AgentRuby-512                    7.332k ± 44%   7.387k ± 67%        ~ (p=0.368 n=6)
geomean                          9.193k         10.23k        +11.28%

@carsonip carsonip requested a review from a team as a code owner August 1, 2023 12:22
@carsonip carsonip changed the title perf: Update vtprotobuf to reuse backing array with referenced structs vtprotobuf: Reuse backing array with referenced structs Aug 1, 2023
@carsonip carsonip changed the title vtprotobuf: Reuse backing array with referenced structs vtprotobuf: Reuse backing array with referenced structs instead of pooling individual objects Aug 1, 2023
tools/go.mod Show resolved Hide resolved
@carsonip carsonip requested review from axw and lahsivjar August 1, 2023 18:28
aggregators/codec.go Outdated Show resolved Hide resolved
Copy link
Member

@axw axw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would like to see the reslicing/indexing to the end of the slices tidied up, but otherwise LGTM!

@carsonip carsonip merged commit 60091c7 into elastic:main Aug 2, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

vtproto UnmarshalVT is allocating since pointer is nil
3 participants