v0.15.1.rc0

What's Changed

• pyo3: update to 0.19 by @mikelui in #1322
• Add expect() for disabling truncation by @boyleconnor in #1316
• Re-using scritpts from safetensors. by @Narsil in #1328
• Reduce number of different revisions by 1 by @Narsil in #1329
• Python 38 arm by @Narsil in #1330
• Move to maturing mimicking move for safetensors. + Rewritten node bindings. by @Narsil in #1331
• Updating the docs with the new command. by @Narsil in #1333
• Update added tokens by @ArthurZucker in #1335
• update package version for dev by @ArthurZucker in #1339
• Added ability to inspect a 'Sequence' pre-tokenizer. by @eaplatanios in #1341
• Let's allow hf_hub < 1.0 by @ArthurZucker in #1344
• Fixing the progressbar. by @Narsil in #1353
• Preparing release. by @Narsil in #1355
• fix a clerical error in the comment by @tiandiweizun in #1356
• fix: remove useless token by @rtrompier in #1371
• Bump @babel/traverse from 7.22.11 to 7.23.2 in /bindings/node by @dependabot in #1370
• Allow hf_hub 0.18 by @mariosasko in #1383
• Allow huggingface_hub<1.0 by @Wauplin in #1385
• [pre_tokenizers] Fix sentencepiece based Metaspace by @ArthurZucker in #1357
• udpate to version = "0.15.1-dev0" by @ArthurZucker in #1390
• Derive Clone on Tokenizer, add Encoding.into_tokens() method by @epwalsh in #1381
• Stale bot. by @Narsil in #1404
• Fix doc links in readme by @Pierrci in #1367
• Faster HF dataset iteration in docs by @mariosasko in #1414
• Add quick doc to byte_level.rs by @steventrouble in #1420
• Fix make bench. by @Narsil in #1428
• Bump follow-redirects from 1.15.1 to 1.15.4 in /tokenizers/examples/unstable_wasm/www by @dependabot in #1430
• pyo3: update to 0.20 by @mikelui in #1386

New Contributors

• @mikelui made their first contribution in #1322
• @eaplatanios made their first contribution in #1341
• @tiandiweizun made their first contribution in #1356
• @rtrompier made their first contribution in #1371
• @mariosasko made their first contribution in #1383
• @Wauplin made their first contribution in #1385
• @steventrouble made their first contribution in #1420

Full Changelog: v0.13.4.rc2...v0.15.1.rc0