feat: FromPretrained to load tokenizer directly from HF#27

Merged

daulet merged 10 commits intodaulet:mainfrom berkayersoyy:feature/hf-download

Nov 7, 2024

+369-4

Commits on Nov 7, 2024

add LoadTokenizerFromHuggingFace function to load tokenizer directly from huggingface, update README.md

berkayersoyy
authored and
daulet
committed
using channels as unbuffered channel, update channel names and minimize some approaches

Resul Berkay Ersoy
authored and
daulet
committed
fix: rename new func to FromPretrained, improve example
daulet
committed
fix: clean up downloadFile
daulet
committed
fix: concurrency issues in case of an error
daulet
committed
fix: make optional params optional
daulet
committed
fix: cache path has to be model specific
daulet
committed
add unit tests for FromPretrained

berkayersoyy
authored and
daulet
committed
migrate to table driven tests, unify/simplify test cases

berkayersoyy
authored and
daulet
committed
fix: clean up nits
daulet
committed