Text Preprocessing
Text preprocessing is an indispensable step for improving analysis accuracy, and consists of three major tasks: character conversion, text normalization, and sentence splitting.
4.1 Character Conversion (Converter)
| Tool | Weekly DL | Total DL | Stars | Description |
|---|---|---|---|---|
| pykakasi | 686k | 45M | 467 | Kana/kanji to romanization conversion |
| cutlet | 31k | 2M | 390 | Japanese to romanization conversion |
| jaconv | 998k | 91M | 352 | Hiragana/katakana/full-width/half-width conversion |
| kanjize | 23k | 2M | 70 | Kanji numeral to/from integer conversion |
jaconv is the most widely used preprocessing tool, with 998k weekly downloads and 91M total downloads. It provides mutual conversion between hiragana, katakana, full-width, and half-width characters in Pure Python.
pykakasi is a romanization tool for kanji-kana mixed text, boasting 686k/week downloads. It also estimates the readings of kanji.
cutlet is a high-precision romanization tool using MeCab. Romanization systems such as Hepburn and Nihon-shiki can be selected.
4.2 Text Normalization (Preprocessor)
| Tool | Weekly DL | Total DL | Stars | Description |
|---|---|---|---|---|
| neologdn | 19k | 2M | 290 | Normalization for mecab-neologd |
| mojimoji | 69k | 14M | 154 | Fast half-width/full-width conversion |
| HojiChar | 917 | 1M | 127 | Pipeline-based preprocessing |
neologdn is a normalization tool often used in combination with the mecab-neologd dictionary, and is effective for unifying variant spellings. Updated in December 2025, active development continues.
HojiChar is a pipeline-based text preprocessing tool for composing and managing multiple preprocessing steps, and is also used for cleaning LLM training data. It records 917/week downloads.
4.3 Sentence Splitting
| Tool | Weekly DL | Total DL | Stars | Description |
|---|---|---|---|---|
| budoux | 57k | 1M | 1.7k | By Google; language-agnostic |
| Bunkai | 238 | 126k | 201 | Sentence boundary disambiguation |
| ja_sentence_segmenter | 4k | 307k | 75 | Simple sentence splitter |
| fast-bunkai | 70 | 11k | 77 | 40-250x faster version of Bunkai |
budoux is a language-agnostic line-break position estimation tool developed by Google, with Stars: 1,700 — far and away the highest in the sentence-splitting category. It adopts a lightweight machine learning-based approach and is also used for line composition in browsers.
Bunkai is a sentence boundary disambiguation tool by Megagon Labs that also handles sentence splitting in cases without punctuation. fast-bunkai is the Rust implementation of Bunkai, achieving a 40-250x speedup.