18

Discussion

The 898 GitHub repositories and 497 Hugging Face repositories surveyed in this study demonstrate the breadth and maturity of the Japanese NLP ecosystem. The following trends are particularly prominent.

Diversification and stability of morphological analysis: The MeCab family of tools maintains its position as the de facto standard, sustaining hundreds of thousands of combined weekly downloads. Meanwhile, diversification of options is progressing, including the growth of SudachiPy (436k/week), the rise of Rust implementations (lindera, vibrato, vaporetto), and the stable use of Pure Python implementations (Janome).

Standardization of UD-compliant parsing: With GiNZA (Stars: 878) at its center, Universal Dependencies-compliant parsing is becoming mainstream. This is making it easier to integrate with multilingual NLP and connect with the international research community.

Vitality in the manga and gaming fields: Tools related to content creation such as manga-ocr (Stars: 2,800), mokuro (Stars: 1,800), and VOICEVOX (combined Stars: 6,300+) have extremely high popularity. This reflects the fusion of Japanese pop culture and NLP technology.

Growth of domain-specific evaluation datasets: Datasets targeting specific domains and social challenges are increasing — including medical (JMedWiC, IgakuQA series), legal (JBE-QA), accessibility (E2U), and LLM-versus-human annotation comparison (annotation-2025). These enable evaluation of practical language understanding that is difficult to measure with general-purpose benchmarks alone.

18.2 Transition Trend from MeCab-centric to Deep Learning

The Japanese NLP technology stack has undergone the following stages of transition:

  1. First generation (before the early 2010s): Pipeline processing using MeCab + CaboCha/KNP was mainstream. Rule-based and statistical methods were central.
  2. Second generation (late 2010s): Introduction of distributed representations such as Word2Vec and ELMo. With the emergence of BERT (2018), Japanese pretrained models such as Tohoku University BERT were published in succession.
  3. Third generation (early 2020s): Emergence of Japanese GPT-2 and Japanese T5. The transition to Transformer-based end-to-end processing accelerated.
  4. Fourth generation (from 2023 onwards): Explosive spread of LLMs. Launch of domestic LLM projects such as LLM-jp and Swallow. Diversification of ChatGPT applications.

However, in morphological analysis, the advantage of the MeCab family remains significant, and traditional methods tend to be preferred in large-scale text processing due to the speed constraints of deep learning methods.

18.3 Analysis of the Hugging Face Ecosystem

Analysis of the 282 models and 215 datasets published on Hugging Face reveals the current state of the Japanese NLP ecosystem.

Model distribution: Text-generation (text generation/LLMs) is the most numerous with 89 models, followed by automatic-speech-recognition at 29, fill-mask (BERT family) at 22, and sentence-similarity (sentence embeddings) at 11. Reflecting the LLM era, concentration in the text-generation category is notable.

Characteristics of the top-downloaded models:

  1. wav2vec2-large-xlsr-53-japanese (16M DL) - Speech recognition
  2. japanese-gpt-neox-small (656k DL) - Text generation
  3. ruri-v3-310m (634k DL) - Sentence embeddings
  4. Spark-X2.5-4B-Heretic-jp-gguf (589k DL) - Text generation (GGUF)
  5. LFM2.5-Audio-1.5B-JP-GGUF (506k DL) - Spoken dialogue (audio-to-audio)
  6. plamo-embedding-1b (486k DL) - Sentence embeddings
  7. bert-base-japanese-whole-word-masking (462k DL) - fill-mask

The fact that the top models span speech recognition, LLMs, and embeddings indicates that the use of Japanese NLP is spreading across diverse tasks.

Major development organizations:

  • Tokyo Institute of Technology (Swallow): The organization that has published the most models. Japanese extensions of diverse base models including Llama 3/3.1, Gemma 2, GPT-OSS, and Qwen3
  • NII/LLM-jp: llm-jp-3/llm-jp-4 series, ModernBERT, AnswerCarefully dataset, and more
  • cl-nagoya (Nagoya University): Ruri series (embeddings), rerankers
  • rinna: Early contributions including GPT-NeoX, GPT-2, and CLIP
  • CyberAgent: OpenCALM, Japanese version of Mistral-Nemo
  • ELYZA: Japanese Llama version, ELYZA-tasks-100
  • SB Intuitions: Sarashina, ModernBERT-Ja
  • Preferred Networks: PLaMo translation, PLaMo embeddings
  • hotchpotch: Rerankers, datasets (fineweb-2-edu, etc.)

Spread of GGUF quantization: Many LLMs are also published in GGUF format (quantized format for llama.cpp), indicating the large demand for local execution. Many converted versions by mmnga and mmnga-o exist.

18.4 Challenges and Prospects for Japanese NLP in the LLM Era

The main challenges for Japanese NLP in the LLM era are as follows.

Token efficiency problem: Many LLMs use tokenizers centered on English, and the tokenization efficiency for Japanese text is low. Research on Japanese-optimized tokenizers such as llm-jp-tokenizer and compare-ja-tokenizer is underway.

Development of evaluation infrastructure: Japanese LLM evaluation benchmarks such as JGLUE, llm-jp-eval, JMMLU, and swallow-evaluation are being rapidly developed, but there is still room for improvement compared to English.

Quality and quantity of training data: Construction of Japanese training data such as llm-jp-corpus and swallow-corpus is underway, but the amount of data is limited compared to English. HojiChar and other preprocessing tools are being used for data quality management.

Safety evaluation: Safety evaluation datasets for Japanese such as do-not-answer-ja and japanese-toxic-dataset are being developed, but evaluation that takes into account Japanese-specific cultural context is still in its early stages.

18.5 Community and Open-Source Contributions

The Japanese NLP open-source ecosystem is supported by diverse actors.

  • University research labs: Kyoto University (KNP, Juman++, KWDLC, etc.), Tohoku University (BERT, PheMT, etc.), Waseda University (JMMLU, etc.)
  • Corporate research labs: Works Applications (Sudachi family), Megagon Labs (GiNZA, Bunkai, etc.), LINE (DistilBERT), rinna (Japanese pretrained models), CyberAgent AI Lab (CAMERA, etc.), Preferred Networks (plamo-translate)
  • National research institutions: NII/LLM-jp (llm-jp-eval, etc.), National Diet Library (NDLOCR, etc.), NINJAL (UniDic, etc.)
  • Individual developers: MeCab (Taku Kudo), pykakasi, polm (fugashi, cutlet, posuto, etc.), Yasunori Yasuoka (UniDic2UD, esupar, etc.)

In particular, the LLM-jp project is Japan's largest LLM development community centered on the National Institute of Informatics, building a comprehensive ecosystem including evaluation tools (llm-jp-eval), fine-tuning (llm-jp-sft), corpus (llm-jp-corpus), and multimodal evaluation (llm-jp-eval-mm).