simdjson : Parsing gigabytes of JSON per second

JSON is everywhere on the Internet. Servers spend a *lot* of time parsing it. We need a fresh approach. The simdjson library uses commonly available SIMD instructions and microparallel algorithms to parse JSON 2.5x faster than RapidJSON and 25x faster than JSON for Modern C++.

• Fast: Over 2.5x faster than commonly used production-grade JSON parsers.
• Record Breaking Features: Minify JSON at 6 GB/s, validate UTF-8 at 13 GB/s, NDJSON at 3.5 GB/s.
• Easy: First-class, easy to use and carefully documented APIs.
• Beyond DOM: Try the new On Demand API for twice the speed (>4GB/s).
• Strict: Full JSON and UTF-8 validation, lossless parsing. Performance with no compromises.
• Automatic: Selects a CPU-tailored parser at runtime. No configuration needed.
• Reliable: From memory allocation to error handling, simdjson's design avoids surprises.
• Peer Reviewed: Our research appears in venues like VLDB Journal, Software: Practice and Experience.

This library is part of the Awesome Modern C++ list.

Table of Contents

• Quick Start
  • On Demand
• Documentation
• Performance results
• Real-world usage
• Bindings and Ports of simdjson
• About simdjson
• Funding
• Contributing to simdjson
• License

Quick Start

The simdjson library is easily consumable with a single .h and .cpp file.

0. Prerequisites: g++ (version 7 or better) or clang++ (version 6 or better), and a 64-bit system with a command-line shell (e.g., Linux, macOS, freeBSD). We also support programming environments like Visual Studio and Xcode, but different steps are needed.

1. Pull simdjson.h and simdjson.cpp into a directory, along with the sample file twitter.json.

   wget https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.h https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.cpp https://raw.githubusercontent.com/simdjson/simdjson/master/jsonexamples/twitter.json
   

2. Create quickstart.cpp:

   #include "simdjson.h"
   int main(void) {
     simdjson::dom::parser parser;
     simdjson::dom::element tweets = parser.load("twitter.json");
     std::cout << tweets["search_metadata"]["count"] << " results." << std::endl;
   }

3. c++ -o quickstart quickstart.cpp simdjson.cpp

4. ./quickstart

   100 results.
   

On Demand

The new On Demand JSON parser is just as easy, but much faster due to just-in-time parsing. It is in alpha right now. More information can be found in the On Demand Guide.

1. Do step 1 of the Quick Start.

2. Create quickstart.cpp:

   #include "simdjson.h"
   using namespace simdjson;
   int main(void) {
      ondemand::parser parser;
      padded_string json = padded_string::load("twitter.json");
      ondemand::document tweets = parser.iterate(json);
      std::cout << uint64_t(tweets["search_metadata"]["count"]) << " results." << std::endl;
   }

3. c++ -march=native -o quickstart quickstart.cpp simdjson.cpp

4. ./quickstart

   100 results.
   

You'll notice that the code here is very similar to the main Quick Start code (and indeed, it does the same thing). However, if you compare the performance, you should find On Demand much faster.

Documentation

Usage documentation is available:

• Basics is an overview of how to use simdjson and its APIs.
• Performance shows some more advanced scenarios and how to tune for them.
• Implementation Selection describes runtime CPU detection and how you can work with it.
• API contains the automatically generated API documentation.

Performance results

The simdjson library uses three-quarters less instructions than state-of-the-art parser RapidJSON and fifty percent less than sajson. To our knowledge, simdjson is the first fully-validating JSON parser to run at gigabytes per second (GB/s) on commodity processors. It can parse millions of JSON documents per second on a single core.

The following figure represents parsing speed in GB/s for parsing various files on an Intel Skylake processor (3.4 GHz) using the GNU GCC 9 compiler (with the -O3 flag). We compare against the best and fastest C++ libraries. The simdjson library offers full unicode (UTF-8) validation and exact number parsing. The RapidJSON library is tested in two modes: fast and exact number parsing. The sajson library offers fast (but not exact) number parsing and partial unicode validation. In this data set, the file sizes range from 65KB (github_events) all the way to 3.3GB (gsoc-2018). Many files are mostly made of numbers: canada, mesh.pretty, mesh, random and numbers: in such instances, we see lower JSON parsing speeds due to the high cost of number parsing. The simdjson library uses exact number parsing which is particular taxing.

On a Skylake processor, the parsing speeds (in GB/s) of various processors on the twitter.json file are as follows, using again GNU GCC 9.1 (with the -O3 flag). The popular JSON for Modern C++ library is particularly slow: it obviously trades parsing speed for other desirable features.

parser	GB/s
simdjson	2.5
RapidJSON UTF8-validation	0.29
RapidJSON UTF8-valid., exact numbers	0.28
RapidJSON insitu, UTF8-validation	0.41
RapidJSON insitu, UTF8-valid., exact	0.39
sajson (insitu, dynamic)	0.62
sajson (insitu, static)	0.88
dropbox	0.13
fastjson	0.27
gason	0.59
ultrajson	0.34
jsmn	0.25
cJSON	0.31
JSON for Modern C++ (nlohmann/json)	0.11

The simdjson library offers high speed whether it processes tiny files (e.g., 300 bytes) or larger files (e.g., 3MB). The following plot presents parsing speed for synthetic files over various sizes generated with a script on a 3.4 GHz Skylake processor (GNU GCC 9, -O3).

All our experiments are reproducible.

You can go beyond 4 GB/s with our new On Demand API. For NDJSON files, we can exceed 3 GB/s with our multithreaded parsing functions.

Real-world usage

• Microsoft FishStore
• Yandex ClickHouse
• Clang Build Analyzer
• Shopify HeapProfiler

If you are planning to use simdjson in a product, please work from one of our releases.

Bindings and Ports of simdjson

We distinguish between "bindings" (which just wrap the C++ code) and a port to another programming language (which reimplements everything).

• ZippyJSON: Swift bindings for the simdjson project.
• libpy_simdjson: high-speed Python bindings for simdjson using libpy.
• pysimdjson: Python bindings for the simdjson project.
• simdjson-rs: Rust port.
• simdjson-rust: Rust wrapper (bindings).
• SimdJsonSharp: C# version for .NET Core (bindings and full port).
• simdjson_nodejs: Node.js bindings for the simdjson project.
• simdjson_php: PHP bindings for the simdjson project.
• simdjson_ruby: Ruby bindings for the simdjson project.
• fast_jsonparser: Ruby bindings for the simdjson project.
• simdjson-go: Go port using Golang assembly.
• rcppsimdjson: R bindings.
• simdjson_erlang: erlang bindings.

About simdjson

The simdjson library takes advantage of modern microarchitectures, parallelizing with SIMD vector instructions, reducing branch misprediction, and reducing data dependency to take advantage of each CPU's multiple execution cores.

Some people enjoy reading our paper: A description of the design and implementation of simdjson is in our research article:

• Geoff Langdale, Daniel Lemire, Parsing Gigabytes of JSON per Second, VLDB Journal 28 (6), 2019.

We have an in-depth paper focused on the UTF-8 validation:

• John Keiser, Daniel Lemire, Validating UTF-8 In Less Than One Instruction Per Byte, Software: Practice & Experience (to appear)

We also have an informal blog post providing some background and context.

For the video inclined,

(it was the best voted talk, we're kinda proud of it).

Funding

The work is supported by the Natural Sciences and Engineering Research Council of Canada under grant number RGPIN-2017-03910.

Contributing to simdjson

Head over to CONTRIBUTING.md for information on contributing to simdjson, and HACKING.md for information on source, building, and architecture/design.

License

This code is made available under the Apache License 2.0.

Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it under the liberal (business-friendly) MIT license.

For compilers that do not support C++17, we bundle the string-view library which is published under the Boost license (http://www.boost.org/LICENSE_1_0.txt). Like the Apache license, the Boost license is a permissive license allowing commercial redistribution.