[Feat] CxS: Set benchmark name and correct eval func #25
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KoyamaSohei
force-pushed
the
skoyama/patch-2
branch
2 times, most recently
from
f3a01d9 to
d45f444
Compare
Signed-off-by: Sohei Koyama <skoyama@ddn.com>
KoyamaSohei
force-pushed
the
skoyama/patch-2
branch
from
d45f444 to
06f37e3
Compare
Contributor
Author
|
@Siddhant-Ray @Shaoting-Feng |
Siddhant-Ray
approved these changes
Jun 8, 2025
Collaborator
Siddhant-Ray
left a comment
Siddhant-Ray
left a comment
There was a problem hiding this comment.
Looks great, thanks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi all,
We’ve renamed the benchmark to CxS — pronounced “six”.
We’ve also changed the scoring function to use the harmonic mean of$C$ and $S$ , rather than maximizing their product.
This is because simply using C×S allows for pathological cases — for example, setting C = 1 and letting S grow arbitrarily large, as long as prefill cost is small relative to compute capacity. In such scenarios, recomputing the prefill each time doesn’t worsen TTFT, making the metric misleading.
We believe the harmonic mean better captures the balanced scalability we aim to measure.
Thanks again, and looking forward to your feedback!