Skip to content

Commit

Permalink
chunk: add fast path to join chunk data (#4765)
Browse files Browse the repository at this point in the history
**Which issue(s) this PR fixes**: 
Fixes #

**What this PR does / why we need it**:
This PR will add fast path to join chunk data using `Array#join`.
`Array#join` is faster and consumes less memory.

* verify
```ruby
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'benchmark-ips'
  gem 'benchmark-memory'
end

def benchmarks(x, ary)
  x.report("b") {
    adding = ''.b
    ary.each do |a|
      adding << a.b
    end
  }

  x.report("join") {
    ary.join.force_encoding(Encoding::ASCII_8BIT)
  }
end

ary = ('あ'..'ん').to_a

Benchmark.ips do |x|
  benchmarks(x, ary)
  x.compare!
end

Benchmark.memory do |x|
  benchmarks(x, ary)
end
```

```
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux]
Warming up --------------------------------------
                   b    15.571k i/100ms
                join    94.632k i/100ms
Calculating -------------------------------------
                   b    153.486k (± 0.9%) i/s    (6.52 μs/i) -    778.550k in   5.072851s
                join    910.273k (± 2.1%) i/s    (1.10 μs/i) -      4.637M in   5.096382s

Comparison:
                join:   910273.4 i/s
                   b:   153486.4 i/s - 5.93x  slower

Calculating -------------------------------------
                   b     3.616k memsize (     0.000  retained)
                        84.000  objects (     0.000  retained)
                        50.000  strings (     0.000  retained)
                join   320.000  memsize (     0.000  retained)
                         1.000  objects (     0.000  retained)
                         1.000  strings (     0.000  retained)
```

**Docs Changes**:

**Release Note**:

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
  • Loading branch information
Watson1978 authored Jan 14, 2025
1 parent 43437e6 commit 6df9954
Showing 1 changed file with 10 additions and 3 deletions.
13 changes: 10 additions & 3 deletions lib/fluent/plugin/buffer/chunk.rb
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,16 @@ def modified_at
# data is array of formatted record string
def append(data, **kwargs)
raise ArgumentError, '`compress: gzip` can be used for Compressable module' if kwargs[:compress] == :gzip
adding = ''.b
data.each do |d|
adding << d.b
begin
adding = data.join.force_encoding(Encoding::ASCII_8BIT)
rescue
# Fallback
# Array#join throws an exception if data contains strings with a different encoding.
# Although such cases may be rare, it should be considered as a safety precaution.
adding = ''.force_encoding(Encoding::ASCII_8BIT)
data.each do |d|
adding << d.b
end
end
concat(adding, data.size)
end
Expand Down

0 comments on commit 6df9954

Please sign in to comment.