Skip to content

Issues: mlcommons/modelbench

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

179 Open 178 Closed
179 Open 178 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Fix cache corruption bug
#785 opened Jan 9, 2025 by wpietri
Ensemble tests are registered in modelgauge
#782 opened Jan 8, 2025 by bkorycki
Improve how locales and prompt file name interact with each other
#781 opened Jan 7, 2025 by rogthefrog
@rogthefrog
Fix modelbench cache bug bug Something isn't working
#778 opened Jan 6, 2025 by bkorycki
fix the smoke tests
#777 opened Jan 2, 2025 by wpietri
Replace Locale object with just strings
#764 opened Dec 18, 2024 by rogthefrog
@rogthefrog
2
User bug: missing secrets for private modellab files when running public benchmark bug Something isn't working
#762 opened Dec 18, 2024 by bkorycki
2
Fix smoke test
#761 opened Dec 17, 2024 by rogthefrog
@rogthefrog
3
Make it obvious if annotator is not deployed successfully
#760 opened Dec 17, 2024 by rogthefrog
1
Rerun 3 SUTs with French prompts
#759 opened Dec 17, 2024 by rogthefrog
@rogthefrog
2
Get rid of SUT wrapper
#753 opened Dec 16, 2024 by bkorycki
Generate shareable report after a benchmark is run
#752 opened Dec 13, 2024 by rogthefrog
Run calibration for French prompts
#750 opened Dec 13, 2024 by rogthefrog
@rogthefrog
1
Fix capitalization of "AI"
#748 opened Dec 13, 2024 by rogthefrog
@rogthefrog
1
Publish results for moderated Mistral SUTs
#746 opened Dec 12, 2024 by bkorycki
Replace static site generator with something simpler
#745 opened Dec 12, 2024 by bkorycki
Remove 0.5 code
#741 opened Dec 10, 2024 by bkorycki
1
Run 3 SUTs with French prompts
#739 opened Dec 10, 2024 by rogthefrog
@rogthefrog
2
Is Mistral moderation code implementable
#738 opened Dec 9, 2024 by bkorycki
Create and run new mistral SUTs with the mitigations provided by mistral
#737 opened Dec 9, 2024 by rogthefrog
Just getting started with this framework, not sure what to do next....
#736 opened Dec 9, 2024 by jolson-ibm
Handle bad JSON in responses more constructively
#734 opened Dec 6, 2024 by rogthefrog
Update disclaimer text referencing version 0.5
#733 opened Dec 6, 2024 by rogthefrog
2
Rerun the official benchmarks after scoring fix
#732 opened Dec 5, 2024 by rogthefrog
@rogthefrog
5
Add grading discrepancy consistency check
#731 opened Dec 5, 2024 by bkorycki
ProTip! Adding no:label will show everything without a label.