
Comparison code of various tokenizers
Description in other languages
Statistics
Statistics as of Nov 17, 2026
GitHub stars
4
±0 (Oct 13〜)
Last commit
Mar 2023
Created on GitHub
Oct 28, 2021
Listed
Oct 27, 2022
Stars over time
Show as a table
| Month | Stars |
|---|---|
| Dec 2025 | 4 |
| Jan 2026 | 4 |
| Feb 2026 | 4 |
| Mar 2026 | 4 |
| Apr 2026 | 4 |
| May 2026 | 4 |
| Jun 2026 | 4 |
| Jul 2026 | 4 |
| Aug 2026 | 4 |
| Sep 2026 | 4 |
| Oct 2026 | 4 |
| Nov 2026 | 4 |
Last snapshot of each month. Star counts of 1,000 or more are rounded to the nearest 100.
Popular in the same category
linderalindera708A morphological analysis library.
vibratodaac-tools430vibrato: Viterbi-based accelerated tokenizer
vaporettodaac-tools302Vaporetto: Very Accelerated POintwise pREdicTion based TOkenizer
kanpyotogatoga108Japanese Morphological Analyzer written in Rust
goyaLeko84Japanese Morphological Analysis written in Rust
mecab-rstsurai72Safe Rust bindings for mecab a part-of-speech and morphological analyzer library
Resources in similar research fields
toirotaishi-i122A comparison tool of Japanese tokenizersShared: Responsible & Trustworthy NLP・Syntactic Text Processing
TinySegmenterMakershogo8214874A tool for creating a custom learning model for TinySegmenter.Shared: Syntactic Text Processing・Text Segmentation
nlp-recipes-jaupura66Samples codes for natural language processing in JapaneseShared: Syntactic Text Processing・Text Segmentation
react-native-japanese-tokenizercraftzdog40Async Japanese Tokenizer Native Plugin for React Native for iOS and AndroidShared: Syntactic Text Processing・Text Segmentation
WordCloud-Japaneseaocattleya10A script that enables morphological analysis-like display of Japanese sentences in WordCloud without using Mecab (a morphological analysis engine).Shared: Syntactic Text Processing・Text Segmentation
chirptextletuananh7ChirpText is a collection of text processing tools for Python.Shared: Syntactic Text Processing・Text Segmentation