04

Text Preprocessing

Text preprocessing is an indispensable step for improving analysis accuracy, and consists of three major tasks: character conversion, text normalization, and sentence splitting.

4.1 Character Conversion (Converter)

Tool Weekly DL Total DL Stars Description
pykakasi 686k 45M 467 Kana/kanji to romanization conversion
cutlet 31k 2M 390 Japanese to romanization conversion
jaconv 998k 91M 352 Hiragana/katakana/full-width/half-width conversion
kanjize 23k 2M 70 Kanji numeral to/from integer conversion

jaconv is the most widely used preprocessing tool, with 998k weekly downloads and 91M total downloads. It provides mutual conversion between hiragana, katakana, full-width, and half-width characters in Pure Python.

pykakasi is a romanization tool for kanji-kana mixed text, boasting 686k/week downloads. It also estimates the readings of kanji.

cutlet is a high-precision romanization tool using MeCab. Romanization systems such as Hepburn and Nihon-shiki can be selected.

pykakasi
Kana/kanji to romanization conversion
467 686k/w
cutlet
Japanese to romanization conversion
390 31k/w
jaconv
Hiragana/katakana/full-width/half-width conversion
352 998k/w
kanjize
Kanji numeral to/from integer conversion
70 23k/w

4.2 Text Normalization (Preprocessor)

Tool Weekly DL Total DL Stars Description
neologdn 19k 2M 290 Normalization for mecab-neologd
mojimoji 69k 14M 154 Fast half-width/full-width conversion
HojiChar 917 1M 127 Pipeline-based preprocessing

neologdn is a normalization tool often used in combination with the mecab-neologd dictionary, and is effective for unifying variant spellings. Updated in December 2025, active development continues.

HojiChar is a pipeline-based text preprocessing tool for composing and managing multiple preprocessing steps, and is also used for cleaning LLM training data. It records 917/week downloads.

neologdn
Normalization for mecab-neologd
290 19k/w
mojimoji
Fast half-width/full-width conversion
154 69k/w
HojiChar
Pipeline-based preprocessing
127 917/w

4.3 Sentence Splitting

Tool Weekly DL Total DL Stars Description
budoux 57k 1M 1.7k By Google; language-agnostic
Bunkai 238 126k 201 Sentence boundary disambiguation
ja_sentence_segmenter 4k 307k 75 Simple sentence splitter
fast-bunkai 70 11k 77 40-250x faster version of Bunkai

budoux is a language-agnostic line-break position estimation tool developed by Google, with Stars: 1,700 — far and away the highest in the sentence-splitting category. It adopts a lightweight machine learning-based approach and is also used for line composition in browsers.

Bunkai is a sentence boundary disambiguation tool by Megagon Labs that also handles sentence splitting in cases without punctuation. fast-bunkai is the Rust implementation of Bunkai, achieving a 40-250x speedup.


budoux
By Google; language-agnostic
1 57k/w
Bunkai
Sentence boundary disambiguation
201 238/w
ja_sentence_segmenter
Simple sentence splitter
75 4k/w
fast-bunkai
40-250x faster version of Bunkai
77 70/w