Corpus
12.1 POS-tagged and NER Corpora
| Corpus | Stars | Description |
|---|---|---|
| ner-wikipedia-dataset | 143 | Wikipedia NER dataset |
| KWDLC | 87 | Kyoto University Web Document Lead Corpus |
| IOB2Corpus | 61 | NER corpus with IOB2 tags |
| UD_Japanese-GSD | 29 | Google UD Japanese treebank |
| ud_japanese-bccwj | 27 | UD treebank for BCCWJ |
| WikipediaAnnotatedCorpus | 30 | Multi-layer annotated Japanese Wikipedia |
| nijl-manyoshutei | 4 | TEI/XML data for the Hirose Hon Man'yoshu held by Kansai University (CC-BY) |
ner-wikipedia-dataset is a Japanese NER dataset published by Stockmark, and is the most widely used in the NER corpus category at Stars: 143.
KWDLC is the Kyoto University Web Document Lead Corpus, annotated with dependency structure, case relations, and coreference. Stars: 87.
The Universal Dependencies Japanese corpora include multiple treebanks: UD_Japanese-GSD (Google UD), UD_Japanese-PUD (Parallel UD), UD_Japanese-GSDLUW (long unit word version), and ud_japanese-bccwj (BCCWJ version).
nijl-manyoshutei (National Institute of Japanese Literature) releases TEI/XML data of the Hirose Hon Man'yoshu collection held by Kansai University under a CC-BY license. It is useful for digital humanities and annotation research on classical Japanese texts. Stars: 4.
12.2 Parallel Corpora
| Corpus | Stars | Description |
|---|---|---|
| JESC | 91 | Large-scale English-Japanese subtitle corpus |
| small_parallel_enja | 97 | 50k English-Japanese parallel (for benchmarking) |
| cjk-compsci-terms | 160 | CJK computer science term glossary |
| Laboro-ParaCorpus | 19 | Japanese-English parallel corpus creation scripts |
| jhpt | 3 | Historical Japanese text with modern Japanese translation |
JESC (Japanese-English Subtitle Corpus) is a large-scale Japanese-English subtitle corpus published by Stanford University, and is one of the standard resources for machine translation research. Stars: 91.
jhpt (nict-astrec-att) is a parallel corpus that aligns original historical Japanese text with modern Japanese translations at the segment level. It can be used for classical-to-modern Japanese translation research. Stars: 3.
12.3 Dialogue Corpora
| Corpus | Stars | Description |
|---|---|---|
| open2ch-dialogue-corpus | 103 | Dialogue corpus from open 2channel |
| llm-japanese-dataset | 91 | Chat data for LLM construction |
| BSD | 77 | Business Scene Dialogue corpus |
| japanese-daily-dialogue | 62 | Daily dialogue dataset (5 topics) |
| JMRD | 30 | Movie recommendation dialogue dataset |
| JMultiWOZ-TC | 4 | Multi-turn dialogue agent Function Calling evaluation |
| HOTATE | 1 | Japanese dialogue dataset with honne/tatemae (inner/outer face) |
| ETCDataset | 14 | Emotion-transcribed conversation dataset (~1,000 dialogues) |
| Novel2DialCorpus | 0 | Method for building chat dialogue corpus from novels |
| diafill-toolkit | 0 | Toolkit for synthesizing filler-rich short-utterance dialogue scripts using LLMs |
llm-japanese-dataset is a Japanese chat dataset for LLM construction, attracting significant attention at Stars: 91, reflecting the recent demand for LLM development.
BSD (Business Scene Dialogue) is a business scene dialogue corpus by the Tsuruoka Laboratory at the University of Tokyo, containing Japanese-English speech and text parallel data. Stars: 77.
JMultiWOZ-TC (llm-jp) is a dataset for evaluating agent Function Calling in multi-turn dialogue scenarios. HOTATE (Ehime University) is a dialogue dataset incorporating honne (inner feelings) and tatemae (public face), concepts unique to Japanese culture. ETCDataset (UEC/Inaba Lab) is a dataset of approximately 1,000 dialogues in which speakers describe their own emotions for each utterance.
diafill-toolkit (SB Intuitions) is a Python toolkit for synthesizing filler-rich (e.g., "um", "uh"), short-utterance Japanese dialogue scripts for speech-based interaction using LLMs. Data generation proceeds in two phases: seed generation (metadata creation) and dialogue generation (script creation).
12.4 Question Answering and Inference Corpora
| Corpus | Stars | Description |
|---|---|---|
| JaQuAD | 111 | Japanese version of SQuAD |
| JSICK | 46 | Japanese sentence entailment and similarity dataset |
| jrte-corpus | 78 | Realistic textual entailment corpus |
| JaNLI | 18 | Adversarial natural language inference dataset |
| jqara | 45 | Question answering dataset for RAG |
| jemhopqa | 30 | Multi-hop QA dataset |
| KUCI | 5 | Kyoto University commonsense inference dataset |
| JBE-QA | 0 | Japanese Bar Exam QA |
| JSTS-Neg | 3 | Japanese semantic similarity dataset for evaluating negation understanding |
| JMedWiC | 4 | Japanese medical domain word-in-context (WiC) dataset |
| E2U | 0 | Dataset for text accessibility transformation (tsutawaru-ka) |
| annotation-2025 | 0 | Dataset comparing human vs. LLM text interpretation annotations |
JaQuAD is a Japanese version of SQuAD published by Skelter Labs, and is the most popular QA dataset at Stars: 111. It is widely used as a benchmark for machine reading comprehension.
JBE-QA (hancules) is a question answering dataset based on the Japanese Bar Exam. It contains legally complex problems requiring professional reasoning ability, making it a useful benchmark for evaluating LLMs' specialist legal knowledge.
JSTS-Neg (reiko-y) is a Japanese semantic textual similarity dataset extending JSTS from JGLUE. It focuses specifically on negation understanding: for example, whether a model correctly identifies that "I like cats" and "I don't like cats" have low semantic similarity. This is a known weakness in many LLMs. Stars: 3.
JMedWiC (Ehime University NLP Group) constructed a word-in-context (WiC) dataset for the Japanese medical domain. For instance, it tests whether a model can tell that "副作用 (side effect)" means drug side effects in one context and economic side effects in another. Stars: 4.
E2U (Social Informatics Research) is a dataset for "tsutawaru-ka" — the process of transforming text into more accessible, easy-to-understand expressions. This relates to plain language Japanese (やさしい日本語) research and accessibility support for people with cognitive or language barriers.
annotation-2025 (Tiny-Colony) provides data for comparing human text interpretations against LLM outputs. It is useful for analyzing where and how LLMs diverge from human understanding in text interpretation tasks.
12.5 Sentiment and Toxicity Datasets
| Dataset | Stars | Description |
|---|---|---|
| wrime | 179 | Subjective/objective sentiment analysis dataset |
| chABSA-dataset | 143 | Aspect-based sentiment analysis |
| japanese-toxic-dataset | 43 | Japanese toxicity schema |
| commonsense-moral-ja | 15 | Commonsense moral judgment dataset |
| do-not-answer-ja | 24 | Safety evaluation dataset (Japanese version) |
WRIME has the highest popularity among sentiment analysis datasets at Stars: 179. Its distinctive feature is that both the writer's subjective emotions and the reader's objective emotions are annotated.
12.6 LLM Training Corpora
| Corpus | Stars | Description |
|---|---|---|
| databricks-dolly-15k-ja | 89 | Japanese translation of Dolly training data |
| alpaca_ja | 86 | Japanese version of the Alpaca dataset |
| oasst1-89k-ja | 16 | Japanese translation of OASST1 |
| llm-jp-corpus | 49 | LLM-jp training corpus |
| instruction_ja | 24 | Japanese instruction data |
| jp-llm-corpus-pii-filter | 7 | Tool for filtering sensitive personal information from LLM training corpora |
In the LLM era, the construction of large-scale Japanese instruction datasets has been actively underway. databricks-dolly-15k-ja and alpaca_ja are Japanese translations of existing English datasets that can be easily used for Japanese LLM fine-tuning.
oasst1-89k-ja is a Japanese translation of the OpenAssistant Conversations Dataset (OASST1), providing human-generated dialogue-format instruction-response data useful for RLHF and instruction tuning. instruction_ja is a curated collection of Japanese instruction data for LLM fine-tuning.
llm-jp-corpus (LLM-jp) is a pretraining corpus purpose-built for the LLM-jp model series. Unlike the translation-based Dolly/Alpaca datasets, it is an independently collected and cleaned corpus of Japanese web text, Wikipedia, and books — better suited for training natural Japanese vocabulary and writing styles. Stars: 49.
jp-llm-corpus-pii-filter (Matsuo Lab) is a Python tool for filtering "sensitive personal information" (medical history, criminal records, etc.) that requires special consideration from LLM training corpora. Stars: 7.
12.7 Speech Corpora
| Corpus | Stars | Description |
|---|---|---|
| reazonspeech | 406 | Large-scale speech corpus built from YouTube |
| jtubespeech | 232 | YouTube Japanese speech corpus |
| ita-corpus | 243 | ITA corpus sentence list |
| koniwa | 61 | Annotated speech collection |
| rohan4600 | 76 | Mora-balanced corpus |
| j-spaw | 6 | Japanese speech corpus for speaker verification and anti-spoofing |
ReazonSpeech is a large-scale Japanese speech corpus built by Reazon Research, with Stars: 406 — the highest popularity among speech corpora. It provides a complete set of tools for building the corpus at home.
JTubeSpeech is a Japanese speech corpus collected from YouTube, with Stars: 232.
J-SpAW (Takamichi Lab) is a Japanese speech corpus for speaker verification and anti-spoofing research. Stars: 6.
12.8 Datasets on Hugging Face
On Hugging Face, 215 Japanese NLP datasets have been published. The datasets with the highest download counts are shown below.
| Dataset | HF DL | Likes | Description |
|---|---|---|---|
| KakologArchives | 74k | 102 | Nico Nico Live comment logs (over 150GB) |
| fineweb-2-edu-japanese (hotchpotch) | 8k | 36 | Japanese web text filtered for educational quality |
| Cauldron-JA (turing-motors) | 7k | 9 | Japanese Vision-Language training data |
| Galgame-VisualNovel-Reupload (joujiboi) | 5k | 41 | Visual novel speech and text |
| Nemotron-Personas-Japan (nvidia) | 5k | 136 | Synthetic Japanese personas |
| ELYZA-tasks-100 (elyza) | 2k | 103 | 100-task Japanese evaluation set |
| Japanese-Eroge-Voice-V2 (NandemoGHS) | 1k | 63 | Japanese speech dataset |
| voicevox-voice-corpus (ayousanz) | 885 | 7 | VOICEVOX synthetic voice data |
| qg_jaquad (lmqg) | 494 | 5 | Question generation data derived from JaQuAD |
| reazon-speech-v2-clone (litagin) | 364 | 12 | ReazonSpeech v2 mirror |
Evaluation Benchmark Datasets
| Dataset | HF DL | Likes | Description |
|---|---|---|---|
| JMTEB (sbintuitions) | 3k | 20 | Embedding evaluation: 5 tasks, 28 datasets |
| JMMMU | 3k | 20 | Multimodal question answering |
| JMedBench (Coldog2333) | 1k | 8 | Medical/biomedical benchmark |
| JMMLU (nlp-waseda) | 1k | 14 | Multitask understanding: 56 subjects, 7,536 questions |
| EDINET-Bench (SakanaAI) | 662 | 16 | Financial benchmark |
| JGLUE (llm-book) | 546 | 16 | Text classification, sentence pair classification, QA |
ELYZA-tasks-100 attracts the highest attention among evaluation datasets with 103 Likes, and is used for comprehensive evaluation of Japanese LLMs through 100 Japanese tasks. Standard benchmarks such as JMTEB and JMMMU (3k each) and JGLUE (546) also see steady use.
Speech Datasets
| Dataset | HF DL | Likes | Description |
|---|---|---|---|
| japanese-anime-speech-v2 (joujiboi) | 3k | 166 | 292k anime speech samples (397.5 hours) |
| japanese-anime-speech (joujiboi) | 946 | 166 | 73k anime speech samples (110 hours) |
| voicevox-voice-corpus (ayousanz) | 885 | 7 | VOICEVOX synthetic voice data |
| reazonspeech (reazon-research) | 771 | 138 | 5 sizes from 8.5h to 35,000h |
| reazon-speech-v2-clone (litagin) | 364 | 12 | ReazonSpeech v2 mirror |
The japanese-anime-speech series has attracted extremely high attention with 166/140 Likes, demonstrating the large demand for anime speech ASR model training.
LLM Training Datasets
| Dataset | HF DL | Likes | Description |
|---|---|---|---|
| databricks-dolly-15k-ja (kunishou) | 1k | 89 | Dolly Japanese translation |
| AnswerCarefully (llm-jp) | 644 | 194 | Data for improving LLM safety |
| llm-japanese-dataset (izumi-lab) | 572 | 148 | 9M+ samples for LoRA training |
| Hachi-Alpaca (HachiML) | 322 | 16 | Quality-verified using Mixtral |
| oasst1-89k-ja (kunishou) | 193 | 26 | OASST1 Japanese translation |
llm-japanese-dataset has the highest popularity among training datasets at 148 Likes, and is an LLM fine-tuning dataset containing more than 9M samples.
Other Notable Datasets
- Nemotron-Personas-Japan (NVIDIA): High attention at 136 Likes. Synthetic Japanese personas based on demographic and geographic data.
- MOMIJI (Turing Motors): 56M Japanese web documents and 249M images. For VLM training.
- auto-wiki-qa (cl-nagoya): Japan's largest free QA dataset with 2.37M pairs.
- ParallelFiction-Ja_En-100k: Parallel data of Japanese web novels and English translations. Likes: 82.