JHumanEval is a hand‑translated Japanese version of the HumanEval benchmark, providing 164 Python programming problems with parallel English and Japanese comments to evaluate Japanese‑LLM code generation while preserving the original English errors.
Description in other languages
Statistics
Statistics as of Nov 16, 2026
Hugging Face downloads (last 30 days)
731
Hugging Face likes
7
Listed
May 13, 2024
Downloads (30-day window) over time
Show as a table
| Month | DL (30d) |
|---|---|
| Dec 2025 | 356 |
| Jan 2026 | 257 |
| Feb 2026 | 212 |
| Mar 2026 | 300 |
| Apr 2026 | 303 |
| May 2026 | 333 |
| Jun 2026 | 411 |
| Jul 2026 | 312 |
| Aug 2026 | 601 |
| Sep 2026 | 406 |
| Oct 2026 | 720 |
| Nov 2026 | 731 |
Downloads over the 30 days before the last snapshot of each month.
Popular in the same category
- JMedBenchColdog23331kJMedBench is a Japanese biomedical LLM benchmark comprising 20 datasets across five tasks (MCQA, NER, STS, etc.) sourced from MedMCQA, PubMedQA, MMLU, and others, each with its own license, and includes a note that translations may contain biases requiring human review.
- vntl-leaderboardlmg-anon857A leaderboard that ranks large language models by their semantic accuracy in translating Japanese visual novels into English, using cosine similarity over 256 samples and reporting chrF scores for comparison with tools like Sugoi Translator, Google Translate, Naver Papago, and Alibaba Translate.
- llm-jp-evalllm-book666Dataset card for the ja‑vicuna‑qa‑benchmark used in the book “Introduction to Large‑Scale LLM II” and created by llm‑jp‑eval for cross‑dataset Japanese LLM evaluation (Apache 2.0).
- EDINET-BenchSakanaAI662EDINET‑Bench is a Japanese financial benchmark that evaluates LLMs on tasks such as accounting fraud detection, earnings forecasting, and industry prediction using ten years of EDINET‑API disclosed reports, with construction and evaluation code provided and the dataset relicensed to PDL 1.0.
- JDocQA-Reasoningricoh-ai269A Japanese benchmark, JDocQA Reasoning Bench, released by Ricoh for the GENIAC project, evaluates LMMs on 1,286 reasoning questions over complex charts, tables, and diagrams using automated LLM-as‑Judge scoring.
- injecagent-jamasahiroid212A Japanese translation of the InjecAgent benchmark (v0.1.1) that preserves original English tool schemas and attack instructions while adding translated agent thoughts, user guidance, and enhanced hacking prefixes for measuring LLM robustness under an MIT license.
Resources in similar research fields
- JMMLUnlp-waseda1kJMMLU is a Japanese Massive Multitask Language Understanding Benchmark featuring 7,536 teacher‑crafted questions across 56 subjects, including professional medicine, psychology, accounting, philosophy, and diverse high‑school disciplines.Shared: Multilinguality・Text Generation
small_parallel_enjaodashi9750k English-Japanese Parallel Corpus for Machine Translation Benchmark.GitHubShared: Multilinguality・Text Generation- databricks-dolly-15k-jakunishou1kAn automatically translated Japanese version of the databricks‑dolly‑15k dataset, licensed CC‑BY‑SA‑3.0 and last updated on 2023‑05‑11.Shared: Multilinguality・Text Generation
jparacrawl-finetuneMorinoseiMorizo104An example usage of JParaCrawl pre-trained Neural Machine Translation (NMT) models.GitHubShared: Multilinguality・Text Generation
koharukoharu-rs6.7kAutomated manga translation tool with LLM, written in Rust.GitHubShared: Multilinguality・Text Generation
shisa-v2shisa-ai35Japanese / English Bilingual LLMGitHubShared: Multilinguality・Text Generation