Skip to content

Issues: mlcommons/modelbench

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

134 Open 177 Closed
134 Open 177 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Update ensemble join method
#667 opened Nov 4, 2024 by bkorycki
Add persona and persona*hazard breakdown to benchmark grading functions
#666 opened Nov 4, 2024 by rogthefrog
operational improvements, round 3
#665 opened Nov 1, 2024 by wpietri
Ensemble annotator aggregation function should be swappable
#663 opened Nov 1, 2024 by rogthefrog
Benchmark scoring function should be swappable
#662 opened Nov 1, 2024 by rogthefrog
any annotation failure means throw out the item
#661 opened Nov 1, 2024 by wpietri
make offline report not terrible
#655 opened Oct 30, 2024 by wpietri
Add Ministral-8B-Instruct-2410
#653 opened Oct 29, 2024 by bkorycki
include final holdback prompt set
#652 opened Oct 29, 2024 by wpietri
Tests get newest dependency versions
#649 opened Oct 28, 2024 by wpietri
add new reference models
#646 opened Oct 28, 2024 by wpietri
calibrate with final reference models
#645 opened Oct 28, 2024 by wpietri
milestone: ready to share final results milestone A goal that other cards add up to
#644 opened Oct 28, 2024 by wpietri
milestone: ready to send final results for validation milestone A goal that other cards add up to
#643 opened Oct 28, 2024 by wpietri
milestone: ready to share practice reports with vendors milestone A goal that other cards add up to
#642 opened Oct 28, 2024 by wpietri
ops-driven improvements
#641 opened Oct 28, 2024 by wpietri
include final practice prompt set
#640 opened Oct 26, 2024 by wpietri
operational improvements, round 2
#639 opened Oct 26, 2024 by wpietri
rehearse the practice test process
#638 opened Oct 26, 2024 by wpietri
journal improvements, round 1
#637 opened Oct 26, 2024 by wpietri
new grading function
#635 opened Oct 25, 2024 by wpietri
@rogthefrog
1
modelbench release
#628 opened Oct 22, 2024 by wpietri
Add switch for practice vs holdback prompts
#620 opened Oct 21, 2024 by wpietri
1
Make private tests available in modelgauge CLI
#619 opened Oct 21, 2024 by bkorycki
Dry run of publishing practice tests
#618 opened Oct 21, 2024 by wpietri
ProTip! Follow long discussions with comments:>50.