Commits on Oct 8, 2024

1. Revamp the benchmark suite …

   There is a large number of outstanding performance PRs that I want to
   merge, but we need a decent benchmark to judge if they are effective.
   
   I went to borrow rapidjson's benchmark suite, which is a good start.
   
   I only kept the comparison with Oj and RapidJSON, because YAJL is
   slower on most benchmarks, so little point comparing to it.
   
   Encoding:
   
   ```
   == Encoding small nested array (121 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json    88.225k i/100ms
                     oj   209.862k i/100ms
              rapidjson   128.978k i/100ms
   Calculating -------------------------------------
                   json    914.611k (± 0.4%) i/s    (1.09 μs/i) -      4.588M in   5.016099s
                     oj      2.163M (± 0.2%) i/s  (462.39 ns/i) -     10.913M in   5.045964s
              rapidjson      1.392M (± 1.3%) i/s  (718.55 ns/i) -      6.965M in   5.005438s
   
   Comparison:
                   json:   914610.6 i/s
                     oj:  2162693.5 i/s - 2.36x  faster
              rapidjson:  1391682.6 i/s - 1.52x  faster
   
   == Encoding small hash (65 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json   142.093k i/100ms
                     oj   651.412k i/100ms
              rapidjson   237.706k i/100ms
   Calculating -------------------------------------
                   json      1.478M (± 0.7%) i/s  (676.78 ns/i) -      7.389M in   5.000866s
                     oj      7.150M (± 0.7%) i/s  (139.85 ns/i) -     35.828M in   5.010756s
              rapidjson      2.250M (± 1.6%) i/s  (444.46 ns/i) -     11.410M in   5.072451s
   
   Comparison:
                   json:  1477595.1 i/s
                     oj:  7150472.0 i/s - 4.84x  faster
              rapidjson:  2249926.7 i/s - 1.52x  faster
   
   == Encoding twitter.json (466906 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json   101.000 i/100ms
                     oj   223.000 i/100ms
              rapidjson   105.000 i/100ms
   Calculating -------------------------------------
                   json      1.017k (± 0.7%) i/s  (982.83 μs/i) -      5.151k in   5.062786s
                     oj      2.244k (± 0.7%) i/s  (445.72 μs/i) -     11.373k in   5.069428s
              rapidjson      1.069k (± 4.6%) i/s  (935.20 μs/i) -      5.355k in   5.016652s
   
   Comparison:
                   json:     1017.5 i/s
                     oj:     2243.6 i/s - 2.21x  faster
              rapidjson:     1069.3 i/s - same-ish: difference falls within error
   
   == Encoding citm_catalog.json (500299 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json    77.000 i/100ms
                     oj   129.000 i/100ms
              rapidjson    96.000 i/100ms
   Calculating -------------------------------------
                   json    767.217 (± 2.5%) i/s    (1.30 ms/i) -      3.850k in   5.021957s
                     oj      1.291k (± 1.5%) i/s  (774.45 μs/i) -      6.579k in   5.096439s
              rapidjson    959.527 (± 1.1%) i/s    (1.04 ms/i) -      4.800k in   5.003052s
   
   Comparison:
                   json:      767.2 i/s
                     oj:     1291.2 i/s - 1.68x  faster
              rapidjson:      959.5 i/s - 1.25x  faster
   
   == Encoding canada.json (2090234 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json     1.000 i/100ms
                     oj     3.000 i/100ms
              rapidjson     1.000 i/100ms
   Calculating -------------------------------------
                   json     19.748 (± 0.0%) i/s   (50.64 ms/i) -     99.000 in   5.013336s
                     oj     31.016 (± 0.0%) i/s   (32.24 ms/i) -    156.000 in   5.029732s
              rapidjson     19.419 (± 0.0%) i/s   (51.50 ms/i) -     98.000 in   5.050382s
   
   Comparison:
                   json:       19.7 i/s
                     oj:       31.0 i/s - 1.57x  faster
              rapidjson:       19.4 i/s - 1.02x  slower
   
   == Encoding many #to_json calls (2661 bytes)
   oj does not match expected output. Skipping
   rapidjson unsupported (Invalid object key type: Object)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json     2.129k i/100ms
   Calculating -------------------------------------
                   json     21.599k (± 0.6%) i/s   (46.30 μs/i) -    108.579k in   5.027198s
   ```
   
   Parsing:
   
   ```
   == Parsing small nested array (121 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json    47.497k i/100ms
                     oj    54.115k i/100ms
              oj strict    53.854k i/100ms
             Oj::Parser   150.904k i/100ms
              rapidjson    80.775k i/100ms
   Calculating -------------------------------------
                   json    481.096k (± 1.1%) i/s    (2.08 μs/i) -      2.422M in   5.035657s
                     oj    554.878k (± 0.6%) i/s    (1.80 μs/i) -      2.814M in   5.071521s
              oj strict    547.888k (± 0.7%) i/s    (1.83 μs/i) -      2.747M in   5.013212s
             Oj::Parser      1.545M (± 0.4%) i/s  (647.16 ns/i) -      7.847M in   5.078302s
              rapidjson    822.422k (± 0.6%) i/s    (1.22 μs/i) -      4.120M in   5.009178s
   
   Comparison:
                   json:   481096.4 i/s
             Oj::Parser:  1545223.5 i/s - 3.21x  faster
              rapidjson:   822422.4 i/s - 1.71x  faster
                     oj:   554877.7 i/s - 1.15x  faster
              oj strict:   547887.7 i/s - 1.14x  faster
   
   == Parsing small hash (65 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json   154.479k i/100ms
                     oj   220.283k i/100ms
              oj strict   249.928k i/100ms
             Oj::Parser   445.062k i/100ms
              rapidjson   289.615k i/100ms
   Calculating -------------------------------------
                   json      1.581M (± 3.0%) i/s  (632.55 ns/i) -      8.033M in   5.086476s
                     oj      2.202M (± 3.5%) i/s  (454.08 ns/i) -     11.014M in   5.008146s
              oj strict      2.498M (± 3.5%) i/s  (400.25 ns/i) -     12.496M in   5.008245s
             Oj::Parser      4.640M (± 0.4%) i/s  (215.50 ns/i) -     23.588M in   5.083443s
              rapidjson      3.111M (± 0.3%) i/s  (321.44 ns/i) -     15.639M in   5.027097s
   
   Comparison:
                   json:  1580898.5 i/s
             Oj::Parser:  4640298.1 i/s - 2.94x  faster
              rapidjson:  3111005.2 i/s - 1.97x  faster
              oj strict:  2498421.4 i/s - 1.58x  faster
                     oj:  2202276.6 i/s - 1.39x  faster
   
   == Parsing test from oj (256 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json    37.580k i/100ms
                     oj    41.899k i/100ms
              oj strict    50.731k i/100ms
             Oj::Parser    74.589k i/100ms
              rapidjson    50.954k i/100ms
   Calculating -------------------------------------
                   json    382.150k (± 1.0%) i/s    (2.62 μs/i) -      1.917M in   5.015737s
                     oj    420.282k (± 0.2%) i/s    (2.38 μs/i) -      2.137M in   5.084338s
              oj strict    511.758k (± 0.5%) i/s    (1.95 μs/i) -      2.587M in   5.055821s
             Oj::Parser    759.087k (± 0.3%) i/s    (1.32 μs/i) -      3.804M in   5.011388s
              rapidjson    518.273k (± 1.8%) i/s    (1.93 μs/i) -      2.599M in   5.015867s
   
   Comparison:
                   json:   382149.6 i/s
             Oj::Parser:   759087.1 i/s - 1.99x  faster
              rapidjson:   518272.8 i/s - 1.36x  faster
              oj strict:   511758.4 i/s - 1.34x  faster
                     oj:   420282.5 i/s - 1.10x  faster
   
   == Parsing twitter.json (567916 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json    52.000 i/100ms
                     oj    63.000 i/100ms
              oj strict    74.000 i/100ms
             Oj::Parser    79.000 i/100ms
              rapidjson    56.000 i/100ms
   Calculating -------------------------------------
                   json    522.896 (± 0.4%) i/s    (1.91 ms/i) -      2.652k in   5.071809s
                     oj    624.849 (± 0.6%) i/s    (1.60 ms/i) -      3.150k in   5.041398s
              oj strict    737.779 (± 0.4%) i/s    (1.36 ms/i) -      3.700k in   5.015117s
             Oj::Parser    789.254 (± 0.3%) i/s    (1.27 ms/i) -      3.950k in   5.004764s
              rapidjson    565.663 (± 0.4%) i/s    (1.77 ms/i) -      2.856k in   5.049015s
   
   Comparison:
                   json:      522.9 i/s
             Oj::Parser:      789.3 i/s - 1.51x  faster
              oj strict:      737.8 i/s - 1.41x  faster
                     oj:      624.8 i/s - 1.19x  faster
              rapidjson:      565.7 i/s - 1.08x  faster
   
   == Parsing citm_catalog.json (1727030 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json    27.000 i/100ms
                     oj    31.000 i/100ms
              oj strict    36.000 i/100ms
             Oj::Parser    42.000 i/100ms
              rapidjson    38.000 i/100ms
   Calculating -------------------------------------
                   json    305.248 (± 0.3%) i/s    (3.28 ms/i) -      1.539k in   5.041813s
                     oj    320.265 (± 3.4%) i/s    (3.12 ms/i) -      1.612k in   5.039715s
              oj strict    373.701 (± 1.6%) i/s    (2.68 ms/i) -      1.872k in   5.010633s
             Oj::Parser    457.792 (± 0.4%) i/s    (2.18 ms/i) -      2.310k in   5.046049s
              rapidjson    350.933 (± 8.8%) i/s    (2.85 ms/i) -      1.748k in   5.052491s
   
   Comparison:
                   json:      305.2 i/s
             Oj::Parser:      457.8 i/s - 1.50x  faster
              oj strict:      373.7 i/s - 1.22x  faster
              rapidjson:      350.9 i/s - 1.15x  faster
                     oj:      320.3 i/s - 1.05x  faster
   
   == Parsing canada.json (2251051 bytes)
   ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
   Warming up --------------------------------------
                   json     2.000 i/100ms
                     oj     2.000 i/100ms
              oj strict     2.000 i/100ms
             Oj::Parser     2.000 i/100ms
              rapidjson    28.000 i/100ms
   Calculating -------------------------------------
                   json     29.216 (± 6.8%) i/s   (34.23 ms/i) -    146.000 in   5.053753s
                     oj     24.899 (± 0.0%) i/s   (40.16 ms/i) -    126.000 in   5.061915s
              oj strict     24.828 (± 4.0%) i/s   (40.28 ms/i) -    124.000 in   5.003067s
             Oj::Parser     30.867 (± 3.2%) i/s   (32.40 ms/i) -    156.000 in   5.057104s
              rapidjson    285.761 (± 1.0%) i/s    (3.50 ms/i) -      1.456k in   5.095715s
   
   Comparison:
                   json:       29.2 i/s
              rapidjson:      285.8 i/s - 9.78x  faster
             Oj::Parser:       30.9 i/s - same-ish: difference falls within error
                     oj:       24.9 i/s - 1.17x  slower
              oj strict:       24.8 i/s - 1.18x  slower
   ```

   

   byroot committed Oct 8, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for d826daa
   • Browse repository at this point

   Copy the full SHA

   d826daa View commit details

   Browse the repository at this point in the history