Nanoscale tokenizer in C++

Nanoscale tokenizer in C++. Currently RWKV world tokenizer is implemented.

Features

Easy to embed
Read vocab from JSON(through minijson)

Variants

Naiive Trie tree implementation : rwkv_world_tokenizer_trie.hh
Efficient version using hat-trie : rwkv_world_tokenizer_hat.hh
Efficient version using cedar : rwkv_world_tokenizer_cedar.hh

If you want to run tokenizer with no C++ exception(e.g. WASM), naiive or cedar version recommended to use.

Additional feature to original RWKV world tokenizer.

UTF-8 byte fallback

TODO

Make C++ Exception free

Third party libraries

minijson : MIT license https://github.com/syoyo/minijson
hat-trie: MIT license https://github.com/Tessil/hat-trie
rwkv_world_tokenizer : Apache 2.0 license https://github.com/mlc-ai/tokenizers-cpp
rwkv_vocab_v20230424.json : Not sure, but would be Apache 2.0 also. https://github.com/BlinkDL/ChatRWKV
cedar/ccedar_core.h : GPLv2, LGPLv2.1 or BSD license. https://www.tkl.iis.u-tokyo.ac.jp/~ynaga/cedar/