About PT

[Henry-Joen]

Dear author, thanks for your hard-working on code updations recently! But I was puzzled about the problem this days, that if I want a model to perform better in a certain field via Chinese, such as the medical field, is it all right that I should do further pre-training on the Chinese-Llama-2 to get the LoRA, instead of on the Chinese-Alpaca-2?
In my understanding, the Chinese-Alpaca-2 is a instructed fine-tuned model via GAtt Method like Llama2-chat, which requires labeled dataset to format inputs between [INST] and [/INST], is thus less sensitive to those unlabeled plain text data.

[ymcui]

1. If you want a foundation model, then you can just tune on top of Chinese-LLaMA with your unlabeled training data.
2. If you want a chat model, you should prepare both unlabeled training data and labeled SFT data. Regarding whether you should tune on top of Chinese-LLaMA or Chinese-Alpaca, there is no strict guideline. You can try both and see which one is better.

[Henry-Joen]

Ok, I know, thanks for your advising! I should have a try later.