
Japanese tokenizer for Transformers
Description in other languages
Statistics
Statistics as of Nov 17, 2026
GitHub stars
81
±0 (Oct 13〜)
Package downloads per week
427
Total package downloads
214,000
Last commit
Dec 2023
Created on GitHub
Jun 22, 2021
Listed
—
Listed since the page was created
Stars over time
Show as a table
| Month | Stars |
|---|---|
| Dec 2025 | 79 |
| Jan 2026 | 79 |
| Feb 2026 | 79 |
| Mar 2026 | 79 |
| Apr 2026 | 79 |
| May 2026 | 79 |
| Jun 2026 | 80 |
| Jul 2026 | 80 |
| Aug 2026 | 81 |
| Sep 2026 | 81 |
| Oct 2026 | 81 |
| Nov 2026 | 81 |
Last snapshot of each month. Star counts of 1,000 or more are rounded to the nearest 100.
Use on PyPI
PyPI packages: sudachitra · Monthly figures to Oct 2026 (ClickPy)
Downloads, last 12 months
64,553
Share from Japan (last 12 months)
14.4%
PyPI packages that depend on it
2
2 as a required dependency
Monthly downloads on PyPI
Show as a table
| Month | downloads (month) |
|---|---|
| Jan 2019 | — |
| Feb 2019 | — |
| Mar 2019 | — |
| Apr 2019 | — |
| May 2019 | — |
| Jun 2019 | — |
| Jul 2019 | — |
| Aug 2019 | — |
| Sep 2019 | — |
| Oct 2019 | — |
| Nov 2019 | — |
| Dec 2019 | — |
| Jan 2020 | — |
| Feb 2020 | — |
| Mar 2020 | — |
| Apr 2020 | — |
| May 2020 | — |
| Jun 2020 | — |
| Jul 2020 | — |
| Aug 2020 | — |
| Sep 2020 | — |
| Oct 2020 | — |
| Nov 2020 | — |
| Dec 2020 | — |
| Jan 2021 | — |
| Feb 2021 | — |
| Mar 2021 | — |
| Apr 2021 | — |
| May 2021 | — |
| Jun 2021 | 138 |
| Jul 2021 | 574 |
| Aug 2021 | 988 |
| Sep 2021 | 1,458 |
| Oct 2021 | 1,553 |
| Nov 2021 | 2,254 |
| Dec 2021 | 3,056 |
| Jan 2022 | 2,551 |
| Feb 2022 | 2,647 |
| Mar 2022 | 3,798 |
| Apr 2022 | 2,314 |
| May 2022 | 1,934 |
| Jun 2022 | 2,660 |
| Jul 2022 | 2,515 |
| Aug 2022 | 2,549 |
| Sep 2022 | 2,636 |
| Oct 2022 | 8,321 |
| Nov 2022 | 3,049 |
| Dec 2022 | 3,204 |
| Jan 2023 | 2,635 |
| Feb 2023 | 2,729 |
| Mar 2023 | 3,498 |
| Apr 2023 | 3,120 |
| May 2023 | 3,013 |
| Jun 2023 | 3,014 |
| Jul 2023 | 3,263 |
| Aug 2023 | 2,997 |
| Sep 2023 | 2,917 |
| Oct 2023 | 2,923 |
| Nov 2023 | 2,565 |
| Dec 2023 | 2,459 |
| Jan 2024 | 2,572 |
| Feb 2024 | 2,614 |
| Mar 2024 | 2,272 |
| Apr 2024 | 2,461 |
| May 2024 | 2,326 |
| Jun 2024 | 2,332 |
| Jul 2024 | 2,779 |
| Aug 2024 | 2,205 |
| Sep 2024 | 2,130 |
| Oct 2024 | 3,381 |
| Nov 2024 | 3,022 |
| Dec 2024 | 2,374 |
| Jan 2025 | 3,082 |
| Feb 2025 | 2,076 |
| Mar 2025 | 2,274 |
| Apr 2025 | 5,509 |
| May 2025 | 1,926 |
| Jun 2025 | 2,401 |
| Jul 2025 | 2,197 |
| Aug 2025 | 4,349 |
| Sep 2025 | 6,223 |
| Oct 2025 | 3,896 |
| Nov 2025 | 3,391 |
| Dec 2025 | 4,960 |
| Jan 2026 | 2,230 |
| Feb 2026 | 1,526 |
| Mar 2026 | 3,681 |
| Apr 2026 | 10,417 |
| May 2026 | 15,231 |
| Jun 2026 | 13,218 |
| Jul 2026 | 3,171 |
| Aug 2026 | 1,801 |
| Sep 2026 | 1,726 |
| Oct 2026 | 3,201 |
Popular in the same category
lukestudio-ousia727LUKE -- Language Understanding with Knowledge-based Embeddings
bert-japanesecl-tohoku551BERT models for Japanese language text.
bert-japaneseyoheikikuta497BERT with SentencePiece for Japanese text.
gpt2-japanesetanreinama322Japanese GPT2 Generation Model
japanese-dialog-transformersnttcslab247Code for evaluating Japanese pretrained models provided by NTT Ltd.
DistilBERT-base-jpBandaiNamcoResearchInc161A Japanese DistilBERT pretrained model, which was trained on Wikipedia.
Resources in similar research fields
jptranstokenizerretarfi5Japanese Tokenizer for transformers libraryShared: Syntactic Text Processing・Text Segmentation
ginza-transformersmegagonlabs16Use custom tokenizers in spacy-transformersShared: Syntactic Text Processing・Text Segmentation
esuparKoichiYasuoka56Tokenizer POS-Tagger and Dependency-parser with BERT/RoBERTa/DeBERTa models for Japanese and other languagesShared: Syntactic Text Processing・Text Segmentation- deberta-v2-large-japanese-char-wwmku-nlp76kJapanese DeBERTa V2 large model trained on 171 GB of Japanese Wikipedia, CC‑100, and OSCAR with character‑level sentencepiece tokenization and whole‑word masking, ready for downstream fine‑tuning through Hugging Face Transformers.HF modelShared: Syntactic Text Processing・Text Segmentation
Mykytea-pythonchezou36Python wrapper for KyTeaShared: Syntactic Text Processing・Text Segmentation
SuPar-UniDicKoichiYasuoka21Tokenizer POS-tagger Lemmatizer and Dependency-parser for modern and contemporary Japanese with BERT modelsShared: Syntactic Text Processing・Text Segmentation