JMedBench is a Japanese biomedical LLM benchmark comprising 20 datasets across five tasks (MCQA, NER, STS, etc.) sourced from MedMCQA, PubMedQA, MMLU, and others, each with its own license, and includes a note that translations may contain biases requiring human review.
Description in other languages
Statistics
Statistics as of Nov 16, 2026
Hugging Face downloads (last 30 days)
1,000
Hugging Face likes
8
Listed
Sep 4, 2024
Downloads (30-day window) over time
Show as a table
| Month | DL (30d) |
|---|---|
| Dec 2025 | 11,000 |
| Jan 2026 | 3,000 |
| Feb 2026 | 879 |
| Mar 2026 | 975 |
| Apr 2026 | 2,000 |
| May 2026 | 5,000 |
| Jun 2026 | 3,000 |
| Jul 2026 | 402 |
| Aug 2026 | 2,000 |
| Sep 2026 | 3,000 |
| Oct 2026 | 2,000 |
| Nov 2026 | 1,000 |
Downloads over the 30 days before the last snapshot of each month.
Popular in the same category
- llm-jp-evalllm-book666Dataset card for the ja‑vicuna‑qa‑benchmark used in the book “Introduction to Large‑Scale LLM II” and created by llm‑jp‑eval for cross‑dataset Japanese LLM evaluation (Apache 2.0).
- JDocQA-Reasoningricoh-ai269A Japanese benchmark, JDocQA Reasoning Bench, released by Ricoh for the GENIAC project, evaluates LMMs on 1,286 reasoning questions over complex charts, tables, and diagrams using automated LLM-as‑Judge scoring.
- ChouBunSakanaAI103ChouBun is a Japanese-language long‑context benchmark for LLMs featuring two task types—extractive QA (wiki_qa, edinet_qa) and abstractive summarization (corp_sec_qa, corp_sec_sum)—built from Wikipedia articles, EDINET security reports, and corporate documents, with standardized THUDM/LongBench format and up to 32768‑token contexts.
- JGLUEshunk0313kUpdated JGLUE dataset card and loading script for a Japanese NLP benchmark (created by Yahoo Japan and Waseda University) that covers text classification (MARC‑ja, JCoLA), sentence‑pair classification (JNLI), and QA (JSQuAD, JCommonsenseQA), with releases linked on GitHub and Hugging Face.
- JamC-QAsbintuitions2kJamC‑QA is a bilingual benchmark of multiple‑choice questions spanning eight Japanese‑culture and knowledge categories, with leaderboard metrics comparing state‑of‑the‑art models.
- JMMMU-ProJMMMU1kJMMMU‑Pro is an image‑based Japanese multimodal benchmark created by generating visual questions with a realistic model and human verification, revealing that current open‑source LMMs perform poorly on it and offering a cost‑effective method for future VQA benchmark development.
Resources in similar research fields
jmed-llmsociocom59JMED-LLM: Japanese Medical Evaluation Dataset for Large Language ModelsGitHubShared: Language Models・Semantic Text Processing- japanese-gpt-neox-smallrinna656kA 12‑layer, 768‑hidden Japanese GPT‑NeoX model trained on CC‑100, C4, and Wikipedia, compatible with Huggingface, with an optional toy prefix‑tuning weight that forces each sentence to end with a smiling‑face emoji.HF modelShared: Language Models・Semantic Text Processing
- llm-jp-3-150mllm-jp75kLLM‑jp‑3‑150m—a 150‑M‑parameter Japanese language model from the National Institute of Informatics’ LLM R&D Center—is distributed in Hugging Face Transformers format, requiring torch ≥ 2.3.0, transformers ≥ 4.40.1, accelerate ≥ 0.29.3, flash‑attn ≥ 2.5.8, and was pretrained on Japanese Wikipedia, Common Crawl, WARP/PDF, WARP/HTML, and Kaken data using a unigram byte‑fallback tokenizer.HF modelShared: Language Models・Semantic Text Processing
- deberta-v3-base-japaneseku-nlp13kJapanese DeBERTa V3 base pre‑trained on 540 B tokens from LLM‑jp v1.0, trained with a modified DeBERTa V3 setup, uses a unigram byte‑fallback tokenizer (no morphological analyzer), and is fine‑tuned for JGLUE NLU tasks.HF modelShared: Language Models・Semantic Text Processing
jp-stabletree155JP Language Model Evaluation HarnessGitHubShared: Language Models・Semantic Text Processing- llm-jp-3-1.8bllm-jp3kA collection of Japanese large‑language models (1.8 b to 172 b beta1, with instruct variants) from NII’s R&D Center, packaged in Hugging Face Transformers format and pretrained on a mix of Japanese, English, and Web corpora totalling >1 trillion tokens, requiring torch ≥ 2.3, transformers ≥ 4.40, accelerate ≥ 0.29, and flash‑attn ≥ 2.5.HF modelShared: Language Models・Semantic Text Processing