Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
pkoukk authored Mar 30, 2023
1 parent ce4f215 commit e9fd197
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
# tiktoken-go
OpenAI's tiktoken in Go.
Tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Currently, it only tested in gpt-3.5(cl100k_base).
Currently, it only tested in cl100k_base (which used in gpt-3.5-turbo and gpt4).
But it should work in r50k_base and p50k_edit.
This is a port of the original [tiktoken](https://github.com/openai/tiktoken).

Go 语言版本的 OpenAI 的 tiktoken。
帮你把文本转换成 OpenAI 的模型可以识别的 token。
目前只测试过 gpt-3.5(cl100k_base)。
目前只测试过 cl100k_base (gpt-3.5和gpt-4目前使用的格式) 。
理论上 r50k_base 和 p50k_edit 也能用(我觉得你们也不用).
tiktoken的原项目地址[tiktoken](https://github.com/openai/tiktoken).


0 comments on commit e9fd197

Please sign in to comment.