
Unsupervised text tokenizer for Neural Network-based text generation.
Description in other languages
Statistics
Statistics as of Nov 17, 2026
GitHub stars
12,000
±0 (Oct 13〜)
Last commit
Nov 2026
Created on GitHub
Mar 7, 2017
Listed
—
Listed since the page was created
Stars over time
Show as a table
| Month | Stars |
|---|---|
| Dec 2025 | — |
| Jan 2026 | — |
| Feb 2026 | — |
| Mar 2026 | — |
| Apr 2026 | — |
| May 2026 | — |
| Jun 2026 | — |
| Jul 2026 | — |
| Aug 2026 | — |
| Sep 2026 | 12,000 |
| Oct 2026 | 12,000 |
| Nov 2026 | 12,000 |
Last snapshot of each month. Star counts of 1,000 or more are rounded to the nearest 100.
Use on PyPI
PyPI packages: sentencepiece · Monthly figures to Oct 2026 (ClickPy)
Downloads, last 12 months
389,251,173
Share from Japan (last 12 months)
1.4%
PyPI packages that depend on it
1,824
1,412 as a required dependency
Monthly downloads on PyPI
Show as a table
| Month | downloads (month) |
|---|---|
| Jan 2019 | 54,237 |
| Feb 2019 | 52,540 |
| Mar 2019 | 66,524 |
| Apr 2019 | 54,626 |
| May 2019 | 70,157 |
| Jun 2019 | 63,544 |
| Jul 2019 | 89,424 |
| Aug 2019 | 126,645 |
| Sep 2019 | 165,084 |
| Oct 2019 | 204,413 |
| Nov 2019 | 258,676 |
| Dec 2019 | 335,380 |
| Jan 2020 | 322,894 |
| Feb 2020 | 403,748 |
| Mar 2020 | 516,140 |
| Apr 2020 | 945,214 |
| May 2020 | 1,388,158 |
| Jun 2020 | 1,262,991 |
| Jul 2020 | 1,216,011 |
| Aug 2020 | 1,284,371 |
| Sep 2020 | 1,436,944 |
| Oct 2020 | 1,720,029 |
| Nov 2020 | 2,140,647 |
| Dec 2020 | 1,574,918 |
| Jan 2021 | 1,633,162 |
| Feb 2021 | 1,696,131 |
| Mar 2021 | 2,009,045 |
| Apr 2021 | 2,096,489 |
| May 2021 | 2,258,493 |
| Jun 2021 | 2,612,420 |
| Jul 2021 | 2,788,863 |
| Aug 2021 | 3,180,523 |
| Sep 2021 | 2,721,353 |
| Oct 2021 | 2,664,970 |
| Nov 2021 | 3,017,958 |
| Dec 2021 | 3,302,593 |
| Jan 2022 | 4,070,128 |
| Feb 2022 | 4,492,176 |
| Mar 2022 | 5,408,149 |
| Apr 2022 | 5,505,491 |
| May 2022 | 7,440,345 |
| Jun 2022 | 5,973,223 |
| Jul 2022 | 5,514,434 |
| Aug 2022 | 5,774,297 |
| Sep 2022 | 5,818,315 |
| Oct 2022 | 7,627,259 |
| Nov 2022 | 7,518,865 |
| Dec 2022 | 7,710,800 |
| Jan 2023 | 8,183,460 |
| Feb 2023 | 7,831,852 |
| Mar 2023 | 10,294,963 |
| Apr 2023 | 9,938,795 |
| May 2023 | 10,867,452 |
| Jun 2023 | 10,332,791 |
| Jul 2023 | 9,766,147 |
| Aug 2023 | 10,583,857 |
| Sep 2023 | 10,409,231 |
| Oct 2023 | 10,458,138 |
| Nov 2023 | 10,563,208 |
| Dec 2023 | 12,248,707 |
| Jan 2024 | 12,192,431 |
| Feb 2024 | 13,720,533 |
| Mar 2024 | 19,356,690 |
| Apr 2024 | 22,292,443 |
| May 2024 | 22,706,375 |
| Jun 2024 | 20,147,923 |
| Jul 2024 | 18,349,136 |
| Aug 2024 | 23,368,283 |
| Sep 2024 | 20,961,790 |
| Oct 2024 | 25,373,437 |
| Nov 2024 | 21,862,058 |
| Dec 2024 | 24,339,370 |
| Jan 2025 | 21,173,684 |
| Feb 2025 | 22,356,560 |
| Mar 2025 | 24,144,106 |
| Apr 2025 | 27,809,389 |
| May 2025 | 27,300,818 |
| Jun 2025 | 23,882,691 |
| Jul 2025 | 25,763,592 |
| Aug 2025 | 31,268,505 |
| Sep 2025 | 27,781,517 |
| Oct 2025 | 32,357,929 |
| Nov 2025 | 32,166,375 |
| Dec 2025 | 31,710,186 |
| Jan 2026 | 31,733,398 |
| Feb 2026 | 27,965,186 |
| Mar 2026 | 33,802,909 |
| Apr 2026 | 32,711,268 |
| May 2026 | 37,659,478 |
| Jun 2026 | 32,304,139 |
| Jul 2026 | 35,582,256 |
| Aug 2026 | 36,493,855 |
| Sep 2026 | 27,335,824 |
| Oct 2026 | 29,786,299 |
Popular in the same category
icuunicode-org3.6kThe home of the ICU project source code.
gsego-ego2.8kGo efficient multilingual NLP and text segmentation; support English, Chinese, Japanese and others.
wtpsplitsegment-any-text1.4kToolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.
pySBDnipunsadvilkar943🐍💯pySBD (Python Sentence Boundary Disambiguation) is a rule-based sentence boundary detection that works out-of-the-box.
charabiameilisearch360Library used by Meilisearch to tokenize queries and documents.
Resources in similar research fields
mecabtaku9101.1kYet another Japanese morphological analyzerGitHubShared: Morphological analysis
SudachiWorksApplications1kA Japanese Tokenizer for BusinessGitHubShared: Morphological analysis
kuromoji.jstakuyaa1kJavaScript implementation of Japanese morphological analyzerGitHubShared: Morphological analysis
kagomeikawaha988Self-contained Japanese Morphological Analyzer written in pure GoGitHubShared: Morphological analysis
janomemocobeta922Japanese morphological analysis engine written in pure PythonGitHubShared: Morphological analysis
linderalindera708A morphological analysis library.GitHubShared: Morphological analysis