daac-tools

All

13 repositories

vaporetto
Public
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
nlp rust analyzer segmentation morphological-analysis tokenization japanese tokenizer
Rust
•
Apache License 2.0
•10•235•0•0•Updated Mar 17, 2025Mar 17, 2025
rucrf
Public
Conditional Random Fields implemented in pure Rust
rust machine-learning crf
Rust
•
Apache License 2.0
•3•8•0•0•Updated Mar 17, 2025Mar 17, 2025
python-daachorse
Public
🐎 A fast implementation of the Aho-Corasick algorithm using the compact double-array data structure. (Python wrapper for daachorse)
python search finite-state-machine text-processing aho-corasick double-array substring-matching
Rust
•
Apache License 2.0
•1•16•0•0•Updated Mar 15, 2025Mar 15, 2025
vibrato
Public
🎤 vibrato: Viterbi-based accelerated tokenizer
nlp rust japanese tokenizer segmentation morphological-analysis tokenization
Rust
•
Apache License 2.0
•15•352•6•1•Updated Mar 14, 2025Mar 14, 2025
python-vaporetto
Public
🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.
python nlp rust japanese tokenizer analyzer segmentation morphological-analysis tokenization
Rust
•
Apache License 2.0
•1•20•0•0•Updated Mar 13, 2025Mar 13, 2025
find-simdoc
Public
Finding all pairs of similar documents time- and memory-efficiently
rust similarity-search document-search all-pairs
Rust
•
Apache License 2.0
•3•60•1•0•Updated Mar 13, 2025Mar 13, 2025
crawdad
Public
🦞 Rust library of natural language dictionaries using character-wise double-array tries.
trie data-structures no-std cjk-characters double-array search rust
Rust
•
Apache License 2.0
•2•30•0•0•Updated Jan 13, 2025Jan 13, 2025
daachorse
Public
🐎 A fast implementation of the Aho-Corasick algorithm using the compact double-array data structure in Rust.
search finite-state-machine text-processing aho-corasick no-std double-array substring-matching rust
Rust
•
Apache License 2.0
•14•211•2•2•Updated Dec 29, 2024Dec 29, 2024
python-vibrato
Public
Viterbi-based accelerated tokenizer (Python wrapper)
python nlp tokenizer segmentation morphological-analysis
Rust
•
Apache License 2.0
•1•41•0•0•Updated Sep 4, 2024Sep 4, 2024
trie-match
Public
Fast match expression optimized for string comparison
rust performance no-std double-array
Rust
•
Apache License 2.0
•0•38•0•0•Updated Jan 29, 2024Jan 29, 2024
vaporetto-models
Public
Tokenization models and training scripts for Vaporetto fast tokenizer
Rust
•
Apache License 2.0
•0•1•0•0•Updated May 30, 2023May 30, 2023
include-bytes-zstd
Public
Includes a file with zstd compression in Rust
rust zstd zstandard
Rust
•
Apache License 2.0
•1•12•0•0•Updated Feb 17, 2023Feb 17, 2023
guidelines
Public
Guidelines for daac-tools community
0•0•0•0•Updated Feb 16, 2023Feb 16, 2023