New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

持续预训练阶段的数据配比 #220

Open

zyzyyy123 opened this issue Sep 11, 2024 · 0 comments

zyzyyy123 commented Sep 11, 2024

您好，能否问一下你们在持续预训练的退火阶段，使用的预训练数据是否和前面稳定训练的数据一致？是否有加入或减少部分预训练数据呢？另外，关于退火阶段预训练数据和sft数据的配比问题，是否有更多的实验？感谢！

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment