
This tool automatically evaluates Japanese multi-modal large language models across multiple datasets.
Description in other languages
Statistics
Statistics as of Nov 17, 2026
GitHub stars
44
±0 (Oct 13〜)
Package downloads per week
81
Total package downloads
20,000
Last commit
Apr 2026
Created on GitHub
Jul 9, 2024
Listed
Mar 25, 2025
Stars over time
Show as a table
| Month | Stars |
|---|---|
| Dec 2025 | 40 |
| Jan 2026 | 40 |
| Feb 2026 | 41 |
| Mar 2026 | 41 |
| Apr 2026 | 41 |
| May 2026 | 41 |
| Jun 2026 | 42 |
| Jul 2026 | 43 |
| Aug 2026 | 43 |
| Sep 2026 | 44 |
| Oct 2026 | 44 |
| Nov 2026 | 44 |
Last snapshot of each month. Star counts of 1,000 or more are rounded to the nearest 100.
Use on PyPI
PyPI packages: eval-mm · Monthly figures to Oct 2026 (ClickPy)
Downloads, last 12 months
5,989
Share from Japan (last 12 months)
2.2%
PyPI packages that depend on it
0
0 as a required dependency
Monthly downloads on PyPI
Show as a table
| Month | downloads (month) |
|---|---|
| Jan 2019 | — |
| Feb 2019 | — |
| Mar 2019 | — |
| Apr 2019 | — |
| May 2019 | — |
| Jun 2019 | — |
| Jul 2019 | — |
| Aug 2019 | — |
| Sep 2019 | — |
| Oct 2019 | — |
| Nov 2019 | — |
| Dec 2019 | — |
| Jan 2020 | — |
| Feb 2020 | — |
| Mar 2020 | — |
| Apr 2020 | — |
| May 2020 | — |
| Jun 2020 | — |
| Jul 2020 | — |
| Aug 2020 | — |
| Sep 2020 | — |
| Oct 2020 | — |
| Nov 2020 | — |
| Dec 2020 | — |
| Jan 2021 | — |
| Feb 2021 | — |
| Mar 2021 | — |
| Apr 2021 | — |
| May 2021 | — |
| Jun 2021 | — |
| Jul 2021 | — |
| Aug 2021 | — |
| Sep 2021 | — |
| Oct 2021 | — |
| Nov 2021 | — |
| Dec 2021 | — |
| Jan 2022 | — |
| Feb 2022 | — |
| Mar 2022 | — |
| Apr 2022 | — |
| May 2022 | — |
| Jun 2022 | — |
| Jul 2022 | — |
| Aug 2022 | — |
| Sep 2022 | — |
| Oct 2022 | — |
| Nov 2022 | — |
| Dec 2022 | — |
| Jan 2023 | — |
| Feb 2023 | — |
| Mar 2023 | — |
| Apr 2023 | — |
| May 2023 | — |
| Jun 2023 | — |
| Jul 2023 | — |
| Aug 2023 | — |
| Sep 2023 | — |
| Oct 2023 | — |
| Nov 2023 | — |
| Dec 2023 | — |
| Jan 2024 | — |
| Feb 2024 | — |
| Mar 2024 | — |
| Apr 2024 | — |
| May 2024 | — |
| Jun 2024 | — |
| Jul 2024 | — |
| Aug 2024 | — |
| Sep 2024 | 402 |
| Oct 2024 | 610 |
| Nov 2024 | 1,760 |
| Dec 2024 | 2,059 |
| Jan 2025 | 1,061 |
| Feb 2025 | 752 |
| Mar 2025 | 2,006 |
| Apr 2025 | 1,452 |
| May 2025 | 790 |
| Jun 2025 | 613 |
| Jul 2025 | 871 |
| Aug 2025 | 440 |
| Sep 2025 | 631 |
| Oct 2025 | 395 |
| Nov 2025 | 456 |
| Dec 2025 | 568 |
| Jan 2026 | 647 |
| Feb 2026 | 395 |
| Mar 2026 | 475 |
| Apr 2026 | 507 |
| May 2026 | 564 |
| Jun 2026 | 505 |
| Jul 2026 | 518 |
| Aug 2026 | 551 |
| Sep 2026 | 299 |
| Oct 2026 | 504 |
Popular in the same category
llm-leaderboardwandb95Project of llm evaluation to Japanese tasks
japanese-llm-rankingyuzu-ai50This repository supports YuzuAI's Rakuda leaderboard of Japanese LLMs, which is a Japanese-focused analogue of LMSYS' Vicuna eval.
japanese-llm-roleplay-benchmarkoshizo42This repository was created to evaluate the performance of character role-playing in Japanese LLM.
gpt4-autoevalnorthern-system-service17Script for automatically evaluating language model responses using GPT-4.
IgakuQA119naoto-iwase10Evaluating LLMs on the 119th Japanese Medical Licensing Examination
jp-tl-benchshisa-ai6Anchored Pairwise LLM Evaluation for Bidirectional Japanese-English Translation
Resources in similar research fields
jmed-llmsociocom59JMED-LLM: Japanese Medical Evaluation Dataset for Large Language ModelsShared: Language Models・Semantic Text Processing- japanese-gpt-neox-smallrinna656kA 12‑layer, 768‑hidden Japanese GPT‑NeoX model trained on CC‑100, C4, and Wikipedia, compatible with Huggingface, with an optional toy prefix‑tuning weight that forces each sentence to end with a smiling‑face emoji.HF modelShared: Language Models・Semantic Text Processing
- llm-jp-3-150mllm-jp75kLLM‑jp‑3‑150m—a 150‑M‑parameter Japanese language model from the National Institute of Informatics’ LLM R&D Center—is distributed in Hugging Face Transformers format, requiring torch ≥ 2.3.0, transformers ≥ 4.40.1, accelerate ≥ 0.29.3, flash‑attn ≥ 2.5.8, and was pretrained on Japanese Wikipedia, Common Crawl, WARP/PDF, WARP/HTML, and Kaken data using a unigram byte‑fallback tokenizer.HF modelShared: Language Models・Semantic Text Processing
IgakuQAjungokasai52Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing ExaminationsShared: Language Models・Semantic Text Processing- llm-jp-3-1.8bllm-jp3kA collection of Japanese large‑language models (1.8 b to 172 b beta1, with instruct variants) from NII’s R&D Center, packaged in Hugging Face Transformers format and pretrained on a mix of Japanese, English, and Web corpora totalling >1 trillion tokens, requiring torch ≥ 2.3, transformers ≥ 4.40, accelerate ≥ 0.29, and flash‑attn ≥ 2.5.HF modelShared: Language Models・Semantic Text Processing
- japanese-gpt-1brinna2kA 1.3‑B‑parameter, 24‑layer transformer GPT‑1B, trained on Japanese C4, CC‑100, and Wikipedia, was released by rinna Co. on January 26 2022 and is available under the MIT license.HF modelShared: Language Models・Semantic Text Processing