Speech Processing and TTS
13.1 Speech Recognition
Whisper Japanese Fine-tuning:
- whisper-asr-finetune (Stars: 32): Fine-tuning scripts for Whisper ASR models
- N46Whisper (Stars: 1,700): Whisper-based Japanese subtitle generation tool
- llm-jp-asr (Stars: 9): Speech recognition with LLM-jp replacing Whisper's decoder
- whisper-transcription (Stars: 20): Audio transcription using Whisper
N46Whisper has Stars: 1,700 — extremely high popularity among speech processing tools. It automatically generates Japanese subtitles using OpenAI's Whisper model and is widely used for subtitling video content.
Julius-related:
- dictation-kit (Stars: 166): A Japanese speech recognition kit using Julius
- julius4seg (Stars: 7): Segmentation support using Julius
Japanese ASR Models on Hugging Face
In the speech recognition (ASR) category on Hugging Face, fine-tuned models for Whisper and Wav2Vec2 are well-represented.
| Model | HF DL | Likes | Description |
|---|---|---|---|
| wav2vec2-large-xlsr-53-japanese (jonatasgrosman) | 16M | 119 | Trained on Common Voice/CSS10/JSUT |
| wav2vec2-large-xlsr-japanese-hiragana (vumichien) | 153k | 11 | ASR with hiragana output |
| anime-whisper (litagin) | 33k | 165 | Anime speech specialist; trained on 5,300 hours |
| kotoba-whisper-v2.0 (kotoba-tech) | 25k | 103 | Distilled from Whisper large-v3; 6.3x speed |
| kotoba-whisper-v2.2 (kotoba-tech) | 16k | 134 | Speaker diarization + auto punctuation |
| japanese-wav2vec2-base-rs35kh (reazon-research) | 10k | 3 | Trained on ReazonSpeech v2.0 |
| parakeet-tdt_ctc-0.6b-ja (nvidia) | 7k | 74 | NVIDIA NeMo; punctuated transcription |
| kotoba-whisper-bilingual-v1.0 (kotoba-tech) | 4k | 19 | Japanese-English bilingual ASR, 6.3x speed |
| reazonspeech-nemo-v2 (reazon-research) | 4k | 41 | 619M parameters; handles long audio |
wav2vec2-large-xlsr-53-japanese shows the largest usage in the Hugging Face ASR category at 16M downloads, and ranks #1 across all Japanese NLP models. It is fine-tuned on the Common Voice, CSS10, and JSUT corpora.
The kotoba-whisper series (Kotoba Technologies): A knowledge distillation model from OpenAI Whisper large-v3, achieving a 6.3x speedup while maintaining CER/WER comparable to the teacher model. In v2.2, speaker diarization and automatic punctuation functions have been integrated.
anime-whisper is a specialized ASR model fine-tuned on approximately 5,300 hours of anime-style dialogue speech, attracting 165 Likes. Features include low hallucination and recognition of non-verbal sounds.
parakeet-tdt_ctc-0.6b-ja (NVIDIA): A FastConformer-based ASR model with 0.6B parameters that performs punctuated transcription of Japanese speech.
reazonspeech-nemo-v2 (Reazon Research): A large 619M-parameter ASR model suited for long-form transcription such as lectures and meetings.
Model selection guide: For general-purpose high-speed transcription, use kotoba-whisper (6.3x faster than Whisper); for anime/game audio, use anime-whisper (handles non-verbal sounds with low hallucination); for long-form content, use reazonspeech-nemo-v2; for precision with punctuation, use parakeet-tdt_ctc-0.6b-ja (NVIDIA NeMo).
13.2 Text-to-Speech (TTS)
| Tool | Stars | Description |
|---|---|---|
| VOICEVOX (editor) | 3,300 | Free text-to-speech software |
| voicevox_engine | 1,800 | VOICEVOX speech synthesis engine |
| voicevox_core | 1,200 | VOICEVOX core library (Rust) |
| pyopenjtalk | 261 | Python wrapper for OpenJTalk |
| tacotron2-japanese | 265 | Japanese implementation of Tacotron2 |
| j-moshi | 327 | Full-duplex voice dialogue system |
| jpreprocess | 61 | Rust rewrite of OpenJTalk |
| piper-plus | 240 | Piper TTS with enhanced Japanese support |
VOICEVOX has a combined Stars of over 6,300 (editor: 3,300 + engine: 1,800 + core: 1,200), making it the most popular Japanese speech synthesis tool by far. As a free, medium-quality text-to-speech software, it is widely used in video production and game development.
pyopenjtalk is a Python wrapper for OpenJTalk (a Japanese text-to-speech synthesis engine), showing wide adoption at 14k weekly downloads. pyopenjtalk-plus (Stars: 69, 5k weekly DL): An additional improvement version.
j-moshi is a Japanese full-duplex voice dialogue system by Nagoya University, with Stars: 327. It is an innovative project that realizes real-time voice interaction.
Japanese Speech-related Models on Hugging Face
In the area of speech synthesis and processing, the following models are published on Hugging Face:
- piper-plus-tsukuyomi-chan (ayousanz): A Japanese TTS model fine-tuned on 100 utterances from the Tsukuyomi-chan corpus. Built on a VITS architecture with a WavLM discriminator and A1/A2/A3 prosody features, exported as a 61MB ONNX file. 2k DL, 11 Likes.
- style-bert-vits2-gguf (kevinzhow): GGUF conversion of Style-Bert-VITS2 models (acoustic model and JP-BERT frontend), enabling local GGML inference via TTS.cpp. 1k DL, 3 Likes.
- LFM2.5-Audio-1.5B-JP-GGUF (LiquidAI): Quantized GGUF build of a Japanese audio model that handles both ASR and TTS in a single model, bundled with CLI/Server runners for llama.cpp. 2k DL, 23 Likes.
13.3 Accent Estimation
MARINE (6gsn/marine): Japanese accent estimation based on multi-task learning. Stars: 38.
marine-plus (tsukumijima/marine-plus): Windows-enhanced version of MARINE. Stars: 9.