Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
214 Open 1,798 Closed
214 Open 1,798 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Add --include_package support for external model modules
#3672 opened Apr 3, 2026 by morgenyu Loading…
1
added OGPTX and pubsector tasks
#3671 opened Apr 2, 2026 by Anirbanbhk88 Loading…
fix: local directory with task name no longer shadows registered task
#3670 opened Apr 1, 2026 by nloughl Loading…
1
Added Toksuite Benchmark
#3669 opened Apr 1, 2026 by gsaltintas Loading…
fix: use np.median for correct median aggregation
#3668 opened Apr 1, 2026 by JuhongPark Loading…
5
Add SimpleQA (OpenAI factuality benchmark)
#3667 opened Mar 31, 2026 by sarthakkgupta Loading…
6 tasks done
1
1
AIFQA-393 BLK-007: [vLLM/lm_eval] ChatGLM2/3 tokenizer — empty stop string crash in lm_eval
#3665 opened Mar 30, 2026 by jklawikowski Loading…
2
Allow Task objects to defer dataset download
#3663 opened Mar 29, 2026 by siddhant-rajhans Loading…
1
Add InfiniteBench: long-context evaluation beyond 100K tokens
#3662 opened Mar 29, 2026 by siddhant-rajhans Loading…
1
MMLU Task Name
#3660 opened Mar 29, 2026 by Teddygat0r Loading…
1
fix: fall back to tokenizer.eos_token when decode returns empty string
#3657 opened Mar 27, 2026 by ganeshr10 Loading…
[BUGFIX] Consistent handling of None answers and cache
#3656 opened Mar 26, 2026 by RawthiL Loading…
feat: add optional SymPy equivalence and math_verify to hendrycks_math
#3655 opened Mar 26, 2026 by NezLheimeur Loading…
fix: Reset batch_sizes cache before each _loglikelihood_tokens call
#3654 opened Mar 26, 2026 by nevertmr Loading…
1
[Hendrycks] Fix false negatives and add flexible_match to Hendrycks
#3653 opened Mar 25, 2026 by fxmarty-amd Loading…
2
1
fix: add Phi-3.5-vision support for vllm-vlm model type
#3651 opened Mar 24, 2026 by ganeshr10 Loading…
1
[Task] NEREL-bench
#3650 opened Mar 23, 2026 by bond005 Loading…
fix: migrate pubmedqa to script-less qiaojin/pubmed_qa dataset #3645
#3649 opened Mar 23, 2026 by Ishitha-P Loading…
1
1
Fix chat_template_args handling when enable_thinking is None
#3648 opened Mar 21, 2026 by ranjita-naik Loading…
1
1
fix: zeno_visualize nested output directory discovery
#3647 opened Mar 21, 2026 by komaksym Loading…
1
1
fix: extract \boxed{} from model response in hendrycks_math
#3644 opened Mar 20, 2026 by NezLheimeur Loading…
1
2
[Task] IFBench
#3642 opened Mar 20, 2026 by RawthiL Loading…
1
Fix ruff lint failures in models/__init__.py and huggingface.py
#3641 opened Mar 20, 2026 by dzautner Loading…
1 task
1
fix: preserve chat_template_args when enable_thinking is None
#3640 opened Mar 19, 2026 by NezLheimeur Loading…
1
1
feat: add TRT-LLM backend.
#3628 opened Mar 11, 2026 by Tracin Loading…
1
ProTip! Updated in the last three days: updated:>2026-03-31.