Expanding llama dimensions after creating new sentencepiece model

[joemkwon]

I followed the script to create a sentencepiece model from chinese-only text. I have the merged_tokenizer_hf directory with my special_tokens_map.json, tokenizer_configs, and tokenizer.model as well as merged_tokenizer_sp directory with the .model file. Now I want to change the original model in necessary ways to match increased vocab size. How do I go about doing this? Thank you!

Edit: I've tried model.resize_token_embeddings(tokenizer.vocab_size) and it resulted in the model going from 3 shards (.bin) to 6. Confusing.

Edit2: The model doubled in size because resize_token_embeddings automatically uses float32. Llama uses float16. So I just halved the model before saving and otherwise it works fine.