
A framework for few-shot evaluation of language models.
Description in other languages
Statistics
Statistics as of Nov 17, 2026
GitHub stars
14,000
±0 (Oct 13〜)
Last commit
Sep 2026
Created on GitHub
Aug 28, 2020
Listed
—
Listed since the page was created
Stars over time
Show as a table
| Month | Stars |
|---|---|
| Dec 2025 | — |
| Jan 2026 | — |
| Feb 2026 | — |
| Mar 2026 | — |
| Apr 2026 | — |
| May 2026 | — |
| Jun 2026 | — |
| Jul 2026 | — |
| Aug 2026 | — |
| Sep 2026 | 14,000 |
| Oct 2026 | 14,000 |
| Nov 2026 | 14,000 |
Last snapshot of each month. Star counts of 1,000 or more are rounded to the nearest 100.
Use on PyPI
PyPI packages: lm-eval · Monthly figures to Oct 2026 (ClickPy)
Downloads, last 12 months
14,697,225
Share from Japan (last 12 months)
0.5%
PyPI packages that depend on it
107
32 as a required dependency
Monthly downloads on PyPI
Show as a table
| Month | downloads (month) |
|---|---|
| Jan 2019 | — |
| Feb 2019 | — |
| Mar 2019 | — |
| Apr 2019 | — |
| May 2019 | — |
| Jun 2019 | — |
| Jul 2019 | — |
| Aug 2019 | — |
| Sep 2019 | — |
| Oct 2019 | — |
| Nov 2019 | — |
| Dec 2019 | — |
| Jan 2020 | — |
| Feb 2020 | — |
| Mar 2020 | — |
| Apr 2020 | — |
| May 2020 | — |
| Jun 2020 | — |
| Jul 2020 | — |
| Aug 2020 | — |
| Sep 2020 | — |
| Oct 2020 | — |
| Nov 2020 | — |
| Dec 2020 | — |
| Jan 2021 | — |
| Feb 2021 | — |
| Mar 2021 | — |
| Apr 2021 | — |
| May 2021 | — |
| Jun 2021 | — |
| Jul 2021 | — |
| Aug 2021 | — |
| Sep 2021 | 414 |
| Oct 2021 | 164 |
| Nov 2021 | 191 |
| Dec 2021 | 148 |
| Jan 2022 | 180 |
| Feb 2022 | 230 |
| Mar 2022 | 836 |
| Apr 2022 | 858 |
| May 2022 | 754 |
| Jun 2022 | 1,937 |
| Jul 2022 | 1,665 |
| Aug 2022 | 2,182 |
| Sep 2022 | 1,740 |
| Oct 2022 | 1,480 |
| Nov 2022 | 1,767 |
| Dec 2022 | 2,223 |
| Jan 2023 | 2,104 |
| Feb 2023 | 2,344 |
| Mar 2023 | 5,648 |
| Apr 2023 | 9,387 |
| May 2023 | 23,830 |
| Jun 2023 | 51,009 |
| Jul 2023 | 23,237 |
| Aug 2023 | 20,170 |
| Sep 2023 | 26,593 |
| Oct 2023 | 95,959 |
| Nov 2023 | 124,910 |
| Dec 2023 | 154,547 |
| Jan 2024 | 179,053 |
| Feb 2024 | 176,653 |
| Mar 2024 | 204,908 |
| Apr 2024 | 276,469 |
| May 2024 | 221,429 |
| Jun 2024 | 188,895 |
| Jul 2024 | 241,841 |
| Aug 2024 | 244,215 |
| Sep 2024 | 292,226 |
| Oct 2024 | 307,713 |
| Nov 2024 | 208,222 |
| Dec 2024 | 217,916 |
| Jan 2025 | 303,413 |
| Feb 2025 | 290,015 |
| Mar 2025 | 489,848 |
| Apr 2025 | 485,166 |
| May 2025 | 527,341 |
| Jun 2025 | 564,942 |
| Jul 2025 | 640,563 |
| Aug 2025 | 733,267 |
| Sep 2025 | 719,530 |
| Oct 2025 | 701,817 |
| Nov 2025 | 579,400 |
| Dec 2025 | 540,308 |
| Jan 2026 | 769,633 |
| Feb 2026 | 1,272,019 |
| Mar 2026 | 1,256,972 |
| Apr 2026 | 1,346,427 |
| May 2026 | 1,545,164 |
| Jun 2026 | 1,714,737 |
| Jul 2026 | 1,572,996 |
| Aug 2026 | 1,711,811 |
| Sep 2026 | 1,059,674 |
| Oct 2026 | 1,328,084 |
Popular in the same category
mtebembeddings-benchmark3.4kMTEB: State-of-the-art evaluation of embeddings across languages and modalities.
sacrebleumjpost1.3kReference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons.
xtremegoogle-research650XTREME is a benchmark for the evaluation of the cross-lingual generalization ability of pre-trained multilingual models that covers 40 typologically diverse languages and includes nine tasks.
pawsgoogle-research-datasets572This dataset contains 108,463 human-labeled and 656k noisily labeled pairs that feature the importance of modeling structure, context, and word order information for the problem of paraphrase identification.
massivealexa572Tools and Modeling Code for the MASSIVE dataset.
belebelefacebookresearch339Repo for the Belebele dataset, a massively multilingual reading comprehension dataset.
Resources in similar research fields
- ELYZA-tasks-100elyza2kA 100‑sample Japanese instruction‑tuning evaluation dataset of annotated tasks—ranging from summarization correction and math reasoning to translation, creative generation, and user‑intent understanding—designed for manual or automatic 5‑point rating of fine‑tuned models.HF datasetShared: Corpora & datasets・Evaluation & benchmarks
- JMMLUnlp-waseda1kJMMLU is a Japanese Massive Multitask Language Understanding Benchmark featuring 7,536 teacher‑crafted questions across 56 subjects, including professional medicine, psychology, accounting, philosophy, and diverse high‑school disciplines.HF datasetShared: Corpora & datasets・Evaluation & benchmarks
url-nlp (MGSM)google-research279MGSM is a benchmark of grade-school math problems, proposed in the paper Language models are multilingual chain-of-thought reasoners.Shared: Corpora & datasets・Evaluation & benchmarks
xl-sumcsebuetnlp279This repository contains the code, data, and models of the paper titled "XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages" published in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.Shared: Corpora & datasets・Evaluation & benchmarks- qg_jaquadlmqg494Japanese JaQuAD, a subset of QG‑Bench, provides sentence‑ and paragraph‑level data with highlighted answer tokens for training Japanese question‑generation models, evaluated by BLEU4, METEOR, ROUGE‑L, BERTScore, and MoverScore.HF datasetShared: Corpora & datasets・Evaluation & benchmarks
pfgen-benchpfnet-research110Preferred Generation BenchmarkGitHubShared: Corpora & datasets・Evaluation & benchmarks