12

Corpus

12.1 POS-tagged and NER Corpora

Corpus Stars Description
ner-wikipedia-dataset 143 Wikipedia NER dataset
KWDLC 87 Kyoto University Web Document Lead Corpus
IOB2Corpus 61 NER corpus with IOB2 tags
UD_Japanese-GSD 29 Google UD Japanese treebank
ud_japanese-bccwj 27 UD treebank for BCCWJ
WikipediaAnnotatedCorpus 30 Multi-layer annotated Japanese Wikipedia
nijl-manyoshutei 4 TEI/XML data for the Hirose Hon Man'yoshu held by Kansai University (CC-BY)

ner-wikipedia-dataset is a Japanese NER dataset published by Stockmark, and is the most widely used in the NER corpus category at Stars: 143.

KWDLC is the Kyoto University Web Document Lead Corpus, annotated with dependency structure, case relations, and coreference. Stars: 87.

The Universal Dependencies Japanese corpora include multiple treebanks: UD_Japanese-GSD (Google UD), UD_Japanese-PUD (Parallel UD), UD_Japanese-GSDLUW (long unit word version), and ud_japanese-bccwj (BCCWJ version).

nijl-manyoshutei (National Institute of Japanese Literature) releases TEI/XML data of the Hirose Hon Man'yoshu collection held by Kansai University under a CC-BY license. It is useful for digital humanities and annotation research on classical Japanese texts. Stars: 4.

ner-wikipedia-dataset
Wikipedia NER dataset
143
KWDLC
Kyoto University Web Document Lead Corpus
87
IOB2Corpus
NER corpus with IOB2 tags
61
UD_Japanese-GSD
Google UD Japanese treebank
29
ud_japanese-bccwj
UD treebank for BCCWJ
27
WikipediaAnnotatedCorpus
Multi-layer annotated Japanese Wikipedia
30
nijl-manyoshutei
TEI/XML data for the Hirose Hon Man'yoshu held by Kansai University (CC-BY)
4

12.2 Parallel Corpora

Corpus Stars Description
JESC 91 Large-scale English-Japanese subtitle corpus
small_parallel_enja 97 50k English-Japanese parallel (for benchmarking)
cjk-compsci-terms 160 CJK computer science term glossary
Laboro-ParaCorpus 19 Japanese-English parallel corpus creation scripts
jhpt 3 Historical Japanese text with modern Japanese translation

JESC (Japanese-English Subtitle Corpus) is a large-scale Japanese-English subtitle corpus published by Stanford University, and is one of the standard resources for machine translation research. Stars: 91.

jhpt (nict-astrec-att) is a parallel corpus that aligns original historical Japanese text with modern Japanese translations at the segment level. It can be used for classical-to-modern Japanese translation research. Stars: 3.

JESC
Large-scale English-Japanese subtitle corpus
91
small_parallel_enja
50k English-Japanese parallel (for benchmarking)
97
cjk-compsci-terms
CJK computer science term glossary
160
Laboro-ParaCorpus
Japanese-English parallel corpus creation scripts
19
jhpt
Historical Japanese text with modern Japanese translation
3

12.3 Dialogue Corpora

Corpus Stars Description
open2ch-dialogue-corpus 103 Dialogue corpus from open 2channel
llm-japanese-dataset 91 Chat data for LLM construction
BSD 77 Business Scene Dialogue corpus
japanese-daily-dialogue 62 Daily dialogue dataset (5 topics)
JMRD 30 Movie recommendation dialogue dataset
JMultiWOZ-TC 4 Multi-turn dialogue agent Function Calling evaluation
HOTATE 1 Japanese dialogue dataset with honne/tatemae (inner/outer face)
ETCDataset 14 Emotion-transcribed conversation dataset (~1,000 dialogues)
Novel2DialCorpus 0 Method for building chat dialogue corpus from novels
diafill-toolkit 0 Toolkit for synthesizing filler-rich short-utterance dialogue scripts using LLMs

llm-japanese-dataset is a Japanese chat dataset for LLM construction, attracting significant attention at Stars: 91, reflecting the recent demand for LLM development.

BSD (Business Scene Dialogue) is a business scene dialogue corpus by the Tsuruoka Laboratory at the University of Tokyo, containing Japanese-English speech and text parallel data. Stars: 77.

JMultiWOZ-TC (llm-jp) is a dataset for evaluating agent Function Calling in multi-turn dialogue scenarios. HOTATE (Ehime University) is a dialogue dataset incorporating honne (inner feelings) and tatemae (public face), concepts unique to Japanese culture. ETCDataset (UEC/Inaba Lab) is a dataset of approximately 1,000 dialogues in which speakers describe their own emotions for each utterance.

diafill-toolkit (SB Intuitions) is a Python toolkit for synthesizing filler-rich (e.g., "um", "uh"), short-utterance Japanese dialogue scripts for speech-based interaction using LLMs. Data generation proceeds in two phases: seed generation (metadata creation) and dialogue generation (script creation).

open2ch-dialogue-corpus
Dialogue corpus from open 2channel
103
llm-japanese-dataset
Chat data for LLM construction
91
BSD
Business Scene Dialogue corpus
77
japanese-daily-dialogue
Daily dialogue dataset (5 topics)
62
JMRD
Movie recommendation dialogue dataset
30
JMultiWOZ-TC
Multi-turn dialogue agent Function Calling evaluation
4
HOTATE
Japanese dialogue dataset with honne/tatemae (inner/outer face)
1
ETCDataset
Emotion-transcribed conversation dataset (~1,000 dialogues)
14
Novel2DialCorpus
Method for building chat dialogue corpus from novels
0
diafill-toolkit
Toolkit for synthesizing filler-rich short-utterance dialogue scripts using LLMs
0

12.4 Question Answering and Inference Corpora

Corpus Stars Description
JaQuAD 111 Japanese version of SQuAD
JSICK 46 Japanese sentence entailment and similarity dataset
jrte-corpus 78 Realistic textual entailment corpus
JaNLI 18 Adversarial natural language inference dataset
jqara 45 Question answering dataset for RAG
jemhopqa 30 Multi-hop QA dataset
KUCI 5 Kyoto University commonsense inference dataset
JBE-QA 0 Japanese Bar Exam QA
JSTS-Neg 3 Japanese semantic similarity dataset for evaluating negation understanding
JMedWiC 4 Japanese medical domain word-in-context (WiC) dataset
E2U 0 Dataset for text accessibility transformation (tsutawaru-ka)
annotation-2025 0 Dataset comparing human vs. LLM text interpretation annotations

JaQuAD is a Japanese version of SQuAD published by Skelter Labs, and is the most popular QA dataset at Stars: 111. It is widely used as a benchmark for machine reading comprehension.

JBE-QA (hancules) is a question answering dataset based on the Japanese Bar Exam. It contains legally complex problems requiring professional reasoning ability, making it a useful benchmark for evaluating LLMs' specialist legal knowledge.

JSTS-Neg (reiko-y) is a Japanese semantic textual similarity dataset extending JSTS from JGLUE. It focuses specifically on negation understanding: for example, whether a model correctly identifies that "I like cats" and "I don't like cats" have low semantic similarity. This is a known weakness in many LLMs. Stars: 3.

JMedWiC (Ehime University NLP Group) constructed a word-in-context (WiC) dataset for the Japanese medical domain. For instance, it tests whether a model can tell that "副作用 (side effect)" means drug side effects in one context and economic side effects in another. Stars: 4.

E2U (Social Informatics Research) is a dataset for "tsutawaru-ka" — the process of transforming text into more accessible, easy-to-understand expressions. This relates to plain language Japanese (やさしい日本語) research and accessibility support for people with cognitive or language barriers.

annotation-2025 (Tiny-Colony) provides data for comparing human text interpretations against LLM outputs. It is useful for analyzing where and how LLMs diverge from human understanding in text interpretation tasks.

JaQuAD
Japanese version of SQuAD
111
JSICK
Japanese sentence entailment and similarity dataset
46
jrte-corpus
Realistic textual entailment corpus
78
JaNLI
Adversarial natural language inference dataset
18
jqara
Question answering dataset for RAG
45
jemhopqa
Multi-hop QA dataset
30
KUCI
Kyoto University commonsense inference dataset
5
JBE-QA
Japanese Bar Exam QA
0
JSTS-Neg
Japanese semantic similarity dataset for evaluating negation understanding
3
JMedWiC
Japanese medical domain word-in-context (WiC) dataset
4
E2U
Dataset for text accessibility transformation (tsutawaru-ka)
0
annotation-2025
Dataset comparing human vs. LLM text interpretation annotations
0

12.5 Sentiment and Toxicity Datasets

Dataset Stars Description
wrime 179 Subjective/objective sentiment analysis dataset
chABSA-dataset 143 Aspect-based sentiment analysis
japanese-toxic-dataset 43 Japanese toxicity schema
commonsense-moral-ja 15 Commonsense moral judgment dataset
do-not-answer-ja 24 Safety evaluation dataset (Japanese version)

WRIME has the highest popularity among sentiment analysis datasets at Stars: 179. Its distinctive feature is that both the writer's subjective emotions and the reader's objective emotions are annotated.

wrime
Subjective/objective sentiment analysis dataset
179
chABSA-dataset
Aspect-based sentiment analysis
143
japanese-toxic-dataset
Japanese toxicity schema
43
commonsense-moral-ja
Commonsense moral judgment dataset
15
do-not-answer-ja
Safety evaluation dataset (Japanese version)
24

12.6 LLM Training Corpora

Corpus Stars Description
databricks-dolly-15k-ja 89 Japanese translation of Dolly training data
alpaca_ja 86 Japanese version of the Alpaca dataset
oasst1-89k-ja 16 Japanese translation of OASST1
llm-jp-corpus 49 LLM-jp training corpus
instruction_ja 24 Japanese instruction data
jp-llm-corpus-pii-filter 7 Tool for filtering sensitive personal information from LLM training corpora

In the LLM era, the construction of large-scale Japanese instruction datasets has been actively underway. databricks-dolly-15k-ja and alpaca_ja are Japanese translations of existing English datasets that can be easily used for Japanese LLM fine-tuning.

oasst1-89k-ja is a Japanese translation of the OpenAssistant Conversations Dataset (OASST1), providing human-generated dialogue-format instruction-response data useful for RLHF and instruction tuning. instruction_ja is a curated collection of Japanese instruction data for LLM fine-tuning.

llm-jp-corpus (LLM-jp) is a pretraining corpus purpose-built for the LLM-jp model series. Unlike the translation-based Dolly/Alpaca datasets, it is an independently collected and cleaned corpus of Japanese web text, Wikipedia, and books — better suited for training natural Japanese vocabulary and writing styles. Stars: 49.

jp-llm-corpus-pii-filter (Matsuo Lab) is a Python tool for filtering "sensitive personal information" (medical history, criminal records, etc.) that requires special consideration from LLM training corpora. Stars: 7.

databricks-dolly-15k-ja
Japanese translation of Dolly training data
89
alpaca_ja
Japanese version of the Alpaca dataset
86
oasst1-89k-ja
Japanese translation of OASST1
16
llm-jp-corpus
LLM-jp training corpus
49
instruction_ja
Japanese instruction data
24
jp-llm-corpus-pii-filter
Tool for filtering sensitive personal information from LLM training corpora
7

12.7 Speech Corpora

Corpus Stars Description
reazonspeech 406 Large-scale speech corpus built from YouTube
jtubespeech 232 YouTube Japanese speech corpus
ita-corpus 243 ITA corpus sentence list
koniwa 61 Annotated speech collection
rohan4600 76 Mora-balanced corpus
j-spaw 6 Japanese speech corpus for speaker verification and anti-spoofing

ReazonSpeech is a large-scale Japanese speech corpus built by Reazon Research, with Stars: 406 — the highest popularity among speech corpora. It provides a complete set of tools for building the corpus at home.

JTubeSpeech is a Japanese speech corpus collected from YouTube, with Stars: 232.

J-SpAW (Takamichi Lab) is a Japanese speech corpus for speaker verification and anti-spoofing research. Stars: 6.

reazonspeech
Large-scale speech corpus built from YouTube
406
jtubespeech
YouTube Japanese speech corpus
232
ita-corpus
ITA corpus sentence list
243
koniwa
Annotated speech collection
61
rohan4600
Mora-balanced corpus
76
j-spaw
Japanese speech corpus for speaker verification and anti-spoofing
6

12.8 Datasets on Hugging Face

On Hugging Face, 215 Japanese NLP datasets have been published. The datasets with the highest download counts are shown below.

Dataset HF DL Likes Description
KakologArchives 74k 102 Nico Nico Live comment logs (over 150GB)
fineweb-2-edu-japanese (hotchpotch) 8k 36 Japanese web text filtered for educational quality
Cauldron-JA (turing-motors) 7k 9 Japanese Vision-Language training data
Galgame-VisualNovel-Reupload (joujiboi) 5k 41 Visual novel speech and text
Nemotron-Personas-Japan (nvidia) 5k 136 Synthetic Japanese personas
ELYZA-tasks-100 (elyza) 2k 103 100-task Japanese evaluation set
Japanese-Eroge-Voice-V2 (NandemoGHS) 1k 63 Japanese speech dataset
voicevox-voice-corpus (ayousanz) 885 7 VOICEVOX synthetic voice data
qg_jaquad (lmqg) 494 5 Question generation data derived from JaQuAD
reazon-speech-v2-clone (litagin) 364 12 ReazonSpeech v2 mirror
KakologArchives
Nico Nico Live comment logs (over 150GB)
74k 102
fineweb-2-edu-japanese (hotchpotch)
Japanese web text filtered for educational quality
8k 36
Cauldron-JA (turing-motors)
Japanese Vision-Language training data
7k 9
Galgame-VisualNovel-Reupload (joujiboi)
Visual novel speech and text
5k 41
Nemotron-Personas-Japan (nvidia)
Synthetic Japanese personas
5k 136
ELYZA-tasks-100 (elyza)
100-task Japanese evaluation set
2k 103
Japanese-Eroge-Voice-V2 (NandemoGHS)
Japanese speech dataset
1k 63
voicevox-voice-corpus (ayousanz)
VOICEVOX synthetic voice data
885 7
qg_jaquad (lmqg)
Question generation data derived from JaQuAD
494 5
reazon-speech-v2-clone (litagin)
ReazonSpeech v2 mirror
364 12

Evaluation Benchmark Datasets

Dataset HF DL Likes Description
JMTEB (sbintuitions) 3k 20 Embedding evaluation: 5 tasks, 28 datasets
JMMMU 3k 20 Multimodal question answering
JMedBench (Coldog2333) 1k 8 Medical/biomedical benchmark
JMMLU (nlp-waseda) 1k 14 Multitask understanding: 56 subjects, 7,536 questions
EDINET-Bench (SakanaAI) 662 16 Financial benchmark
JGLUE (llm-book) 546 16 Text classification, sentence pair classification, QA

ELYZA-tasks-100 attracts the highest attention among evaluation datasets with 103 Likes, and is used for comprehensive evaluation of Japanese LLMs through 100 Japanese tasks. Standard benchmarks such as JMTEB and JMMMU (3k each) and JGLUE (546) also see steady use.

JMTEB (sbintuitions)
Embedding evaluation: 5 tasks, 28 datasets
3k 20
JMMMU
Multimodal question answering
3k 20
JMedBench (Coldog2333)
Medical/biomedical benchmark
1k 8
JMMLU (nlp-waseda)
Multitask understanding: 56 subjects, 7,536 questions
1k 14
EDINET-Bench (SakanaAI)
Financial benchmark
662 16
JGLUE (llm-book)
Text classification, sentence pair classification, QA
546 16

Speech Datasets

Dataset HF DL Likes Description
japanese-anime-speech-v2 (joujiboi) 3k 166 292k anime speech samples (397.5 hours)
japanese-anime-speech (joujiboi) 946 166 73k anime speech samples (110 hours)
voicevox-voice-corpus (ayousanz) 885 7 VOICEVOX synthetic voice data
reazonspeech (reazon-research) 771 138 5 sizes from 8.5h to 35,000h
reazon-speech-v2-clone (litagin) 364 12 ReazonSpeech v2 mirror

The japanese-anime-speech series has attracted extremely high attention with 166/140 Likes, demonstrating the large demand for anime speech ASR model training.

japanese-anime-speech-v2 (joujiboi)
292k anime speech samples (397.5 hours)
3k 166
japanese-anime-speech (joujiboi)
73k anime speech samples (110 hours)
946 166
voicevox-voice-corpus (ayousanz)
VOICEVOX synthetic voice data
885 7
reazonspeech (reazon-research)
5 sizes from 8.5h to 35,000h
771 138
reazon-speech-v2-clone (litagin)
ReazonSpeech v2 mirror
364 12

LLM Training Datasets

Dataset HF DL Likes Description
databricks-dolly-15k-ja (kunishou) 1k 89 Dolly Japanese translation
AnswerCarefully (llm-jp) 644 194 Data for improving LLM safety
llm-japanese-dataset (izumi-lab) 572 148 9M+ samples for LoRA training
Hachi-Alpaca (HachiML) 322 16 Quality-verified using Mixtral
oasst1-89k-ja (kunishou) 193 26 OASST1 Japanese translation

llm-japanese-dataset has the highest popularity among training datasets at 148 Likes, and is an LLM fine-tuning dataset containing more than 9M samples.

databricks-dolly-15k-ja (kunishou)
Dolly Japanese translation
1k 89
AnswerCarefully (llm-jp)
Data for improving LLM safety
644 194
llm-japanese-dataset (izumi-lab)
9M+ samples for LoRA training
572 148
Hachi-Alpaca (HachiML)
Quality-verified using Mixtral
322 16
oasst1-89k-ja (kunishou)
OASST1 Japanese translation
193 26

Other Notable Datasets

  • Nemotron-Personas-Japan (NVIDIA): High attention at 136 Likes. Synthetic Japanese personas based on demographic and geographic data.
  • MOMIJI (Turing Motors): 56M Japanese web documents and 249M images. For VLM training.
  • auto-wiki-qa (cl-nagoya): Japan's largest free QA dataset with 2.37M pairs.
  • ParallelFiction-Ja_En-100k: Parallel data of Japanese web novels and English translations. Likes: 82.

Nemotron-Personas-Japan
- Nemotron-Personas-Japan (NVIDIA): High attention at 136 Likes. Synthetic Japanese personas based on demographic and geographic data.
MOMIJI
- MOMIJI (Turing Motors): 56M Japanese web documents and 249M images. For VLM training.
auto-wiki-qa
- auto-wiki-qa (cl-nagoya): Japan's largest free QA dataset with 2.37M pairs.