v1.0.1
first release
adds the following functionalities:
- BPETokenizer: which can be used to build your tokenizer for the LLM
- Tokenizer: a base class which leverages the save and load of the vocab and merges
first release
adds the following functionalities: