Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

what's the difference for speculative-eagle-topk speculative-num-draft-tokens #3883

Open
WeiboXu opened this issue Feb 26, 2025 · 0 comments
Assignees

Comments

@WeiboXu
Copy link

WeiboXu commented Feb 26, 2025

what's the difference for speculative-eagle-topk speculative-num-draft-tokens ,from the example, by default it will use --speculative-eagle-topk 8 --speculative-num-draft-tokens 64, and from the help ,it means:
--speculative-num-draft-tokens SPECULATIVE_NUM_DRAFT_TOKENS
The number of token sampled from draft model in Speculative Decoding.
--speculative-eagle-topk {1,2,4,8}
The number of token sampled from draft model in eagle2 each step.

My question is:
which one is the value for tokens numbers which generated by draft model first? speculative-num-draft-tokens or speculative-eagle-topk? And what does mean for another one.

Thanks.

@minleminzui minleminzui self-assigned this Feb 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants