13

Speech Processing and TTS

13.1 Speech Recognition

Whisper Japanese Fine-tuning:

  • whisper-asr-finetune (Stars: 32): Fine-tuning scripts for Whisper ASR models
  • N46Whisper (Stars: 1,700): Whisper-based Japanese subtitle generation tool
  • llm-jp-asr (Stars: 9): Speech recognition with LLM-jp replacing Whisper's decoder
  • whisper-transcription (Stars: 20): Audio transcription using Whisper

N46Whisper has Stars: 1,700 — extremely high popularity among speech processing tools. It automatically generates Japanese subtitles using OpenAI's Whisper model and is widely used for subtitling video content.

Julius-related:

  • dictation-kit (Stars: 166): A Japanese speech recognition kit using Julius
  • julius4seg (Stars: 7): Segmentation support using Julius
whisper-asr-finetune
Fine-tuning scripts for Whisper ASR models
32
N46Whisper
Whisper-based Japanese subtitle generation tool
1.7k
llm-jp-asr
Speech recognition with LLM-jp replacing Whisper's decoder
9
whisper-transcription
Audio transcription using Whisper
20
dictation-kit
A Japanese speech recognition kit using Julius
166
julius4seg
Segmentation support using Julius
7

Japanese ASR Models on Hugging Face

In the speech recognition (ASR) category on Hugging Face, fine-tuned models for Whisper and Wav2Vec2 are well-represented.

Model HF DL Likes Description
wav2vec2-large-xlsr-53-japanese (jonatasgrosman) 16M 119 Trained on Common Voice/CSS10/JSUT
wav2vec2-large-xlsr-japanese-hiragana (vumichien) 153k 11 ASR with hiragana output
anime-whisper (litagin) 33k 165 Anime speech specialist; trained on 5,300 hours
kotoba-whisper-v2.0 (kotoba-tech) 25k 103 Distilled from Whisper large-v3; 6.3x speed
kotoba-whisper-v2.2 (kotoba-tech) 16k 134 Speaker diarization + auto punctuation
japanese-wav2vec2-base-rs35kh (reazon-research) 10k 3 Trained on ReazonSpeech v2.0
parakeet-tdt_ctc-0.6b-ja (nvidia) 7k 74 NVIDIA NeMo; punctuated transcription
kotoba-whisper-bilingual-v1.0 (kotoba-tech) 4k 19 Japanese-English bilingual ASR, 6.3x speed
reazonspeech-nemo-v2 (reazon-research) 4k 41 619M parameters; handles long audio

wav2vec2-large-xlsr-53-japanese shows the largest usage in the Hugging Face ASR category at 16M downloads, and ranks #1 across all Japanese NLP models. It is fine-tuned on the Common Voice, CSS10, and JSUT corpora.

The kotoba-whisper series (Kotoba Technologies): A knowledge distillation model from OpenAI Whisper large-v3, achieving a 6.3x speedup while maintaining CER/WER comparable to the teacher model. In v2.2, speaker diarization and automatic punctuation functions have been integrated.

anime-whisper is a specialized ASR model fine-tuned on approximately 5,300 hours of anime-style dialogue speech, attracting 165 Likes. Features include low hallucination and recognition of non-verbal sounds.

parakeet-tdt_ctc-0.6b-ja (NVIDIA): A FastConformer-based ASR model with 0.6B parameters that performs punctuated transcription of Japanese speech.

reazonspeech-nemo-v2 (Reazon Research): A large 619M-parameter ASR model suited for long-form transcription such as lectures and meetings.

Model selection guide: For general-purpose high-speed transcription, use kotoba-whisper (6.3x faster than Whisper); for anime/game audio, use anime-whisper (handles non-verbal sounds with low hallucination); for long-form content, use reazonspeech-nemo-v2; for precision with punctuation, use parakeet-tdt_ctc-0.6b-ja (NVIDIA NeMo).

wav2vec2-large-xlsr-53-japanese (jonatasgrosman)
Trained on Common Voice/CSS10/JSUT
16M 119
wav2vec2-large-xlsr-japanese-hiragana (vumichien)
ASR with hiragana output
153k 11
anime-whisper (litagin)
Anime speech specialist; trained on 5,300 hours
33k 165
kotoba-whisper-v2.0 (kotoba-tech)
Distilled from Whisper large-v3; 6.3x speed
25k 103
kotoba-whisper-v2.2 (kotoba-tech)
Speaker diarization + auto punctuation
16k 134
japanese-wav2vec2-base-rs35kh (reazon-research)
Trained on ReazonSpeech v2.0
10k 3
parakeet-tdt_ctc-0.6b-ja (nvidia)
NVIDIA NeMo; punctuated transcription
7k 74
kotoba-whisper-bilingual-v1.0 (kotoba-tech)
Japanese-English bilingual ASR, 6.3x speed
4k 19
reazonspeech-nemo-v2 (reazon-research)
619M parameters; handles long audio
4k 41
kotoba-whisper series
The kotoba-whisper series (Kotoba Technologies): A knowledge distillation model from OpenAI Whisper large-v3, achieving a 6.3x speedup while maintaining CER/WER comparable to the teacher model. In v2.2, speaker diarization and automatic punctuation functions have been integrated.
parakeet-tdt_ctc-0.6b-ja
parakeet-tdt_ctc-0.6b-ja (NVIDIA): A FastConformer-based ASR model with 0.6B parameters that performs punctuated transcription of Japanese speech.
reazonspeech-nemo-v2
reazonspeech-nemo-v2 (Reazon Research): A large 619M-parameter ASR model suited for long-form transcription such as lectures and meetings.

13.2 Text-to-Speech (TTS)

Tool Stars Description
VOICEVOX (editor) 3,300 Free text-to-speech software
voicevox_engine 1,800 VOICEVOX speech synthesis engine
voicevox_core 1,200 VOICEVOX core library (Rust)
pyopenjtalk 261 Python wrapper for OpenJTalk
tacotron2-japanese 265 Japanese implementation of Tacotron2
j-moshi 327 Full-duplex voice dialogue system
jpreprocess 61 Rust rewrite of OpenJTalk
piper-plus 240 Piper TTS with enhanced Japanese support

VOICEVOX has a combined Stars of over 6,300 (editor: 3,300 + engine: 1,800 + core: 1,200), making it the most popular Japanese speech synthesis tool by far. As a free, medium-quality text-to-speech software, it is widely used in video production and game development.

pyopenjtalk is a Python wrapper for OpenJTalk (a Japanese text-to-speech synthesis engine), showing wide adoption at 14k weekly downloads. pyopenjtalk-plus (Stars: 69, 5k weekly DL): An additional improvement version.

j-moshi is a Japanese full-duplex voice dialogue system by Nagoya University, with Stars: 327. It is an innovative project that realizes real-time voice interaction.

VOICEVOX (editor)
Free text-to-speech software
3.3k
voicevox_engine
VOICEVOX speech synthesis engine
1.8k
voicevox_core
VOICEVOX core library (Rust)
1.2k
pyopenjtalk
Python wrapper for OpenJTalk
261
tacotron2-japanese
Japanese implementation of Tacotron2
265
j-moshi
Full-duplex voice dialogue system
327
jpreprocess
Rust rewrite of OpenJTalk
61
piper-plus
Piper TTS with enhanced Japanese support
240
pyopenjtalk-plus
pyopenjtalk is a Python wrapper for OpenJTalk (a Japanese text-to-speech synthesis engine), showing wide adoption at 14k weekly downloads. pyopenjtalk-plus (Stars: 69, 5k weekly DL): An additional improvement version.
69 14k/w

In the area of speech synthesis and processing, the following models are published on Hugging Face:

  • piper-plus-tsukuyomi-chan (ayousanz): A Japanese TTS model fine-tuned on 100 utterances from the Tsukuyomi-chan corpus. Built on a VITS architecture with a WavLM discriminator and A1/A2/A3 prosody features, exported as a 61MB ONNX file. 2k DL, 11 Likes.
  • style-bert-vits2-gguf (kevinzhow): GGUF conversion of Style-Bert-VITS2 models (acoustic model and JP-BERT frontend), enabling local GGML inference via TTS.cpp. 1k DL, 3 Likes.
  • LFM2.5-Audio-1.5B-JP-GGUF (LiquidAI): Quantized GGUF build of a Japanese audio model that handles both ASR and TTS in a single model, bundled with CLI/Server runners for llama.cpp. 2k DL, 23 Likes.
piper-plus-tsukuyomi-chan
- piper-plus-tsukuyomi-chan (ayousanz): A Japanese TTS model fine-tuned on 100 utterances from the Tsukuyomi-chan corpus. Built on a VITS architecture with a WavLM discriminator and A1/A2/A3 prosody features, exported as a 61MB ONNX file. 2k DL, 11 Likes.
style-bert-vits2-gguf
- style-bert-vits2-gguf (kevinzhow): GGUF conversion of Style-Bert-VITS2 models (acoustic model and JP-BERT frontend), enabling local GGML inference via TTS.cpp. 1k DL, 3 Likes.
LFM2.5-Audio-1.5B-JP-GGUF
- LFM2.5-Audio-1.5B-JP-GGUF (LiquidAI): Quantized GGUF build of a Japanese audio model that handles both ASR and TTS in a single model, bundled with CLI/Server runners for llama.cpp. 2k DL, 23 Likes.

13.3 Accent Estimation

MARINE (6gsn/marine): Japanese accent estimation based on multi-task learning. Stars: 38.

marine-plus (tsukumijima/marine-plus): Windows-enhanced version of MARINE. Stars: 9.


MARINE
MARINE (6gsn/marine): Japanese accent estimation based on multi-task learning. Stars: 38.
38
marine-plus
marine-plus (tsukumijima/marine-plus): Windows-enhanced version of MARINE. Stars: 9.
9