
How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese - ACL SRW 2023
Description in other languages
Statistics
Statistics as of Nov 17, 2026
GitHub stars
6
±0 (Oct 13〜)
Last commit
Jun 2023
Created on GitHub
May 2, 2023
Listed
Jul 18, 2023
Stars over time
Show as a table
| Month | Stars |
|---|---|
| Dec 2025 | 6 |
| Jan 2026 | 6 |
| Feb 2026 | 6 |
| Mar 2026 | 6 |
| Apr 2026 | 6 |
| May 2026 | 6 |
| Jun 2026 | 6 |
| Jul 2026 | 6 |
| Aug 2026 | 6 |
| Sep 2026 | 6 |
| Oct 2026 | 6 |
| Nov 2026 | 6 |
Last snapshot of each month. Star counts of 1,000 or more are rounded to the nearest 100.
Popular in the same category
ginza-transformersmegagonlabs16Use custom tokenizers in spacy-transformers
allennlp-shiba-modelshunk03112AllenNLP integration for Shiba: Japanese CANINE model
jptranstokenizerretarfi5Japanese Tokenizer for transformers library
JGLUEyahoojapan349JGLUE: Japanese General Language Understanding Evaluation
llm-jp-evalllm-jp175This tool is designed to automatically evaluate large-scale Japanese language models across multiple datasets.
jp-stabletree155JP Language Model Evaluation Harness
Resources in similar research fields
toirotaishi-i122A comparison tool of Japanese tokenizersShared: Responsible & Trustworthy NLP・Syntactic Text Processing
SudachiTraWorksApplications81Japanese tokenizer for TransformersShared: Syntactic Text Processing・Text Segmentation
tokenizer-speed-benchlegalforce-research4Comparison code of various tokenizersShared: Responsible & Trustworthy NLP・Syntactic Text Processing
TinySegmenterMakershogo8214874A tool for creating a custom learning model for TinySegmenter.Shared: Syntactic Text Processing・Text Segmentation
nlp-recipes-jaupura66Samples codes for natural language processing in JapaneseShared: Syntactic Text Processing・Text Segmentation
react-native-japanese-tokenizercraftzdog40Async Japanese Tokenizer Native Plugin for React Native for iOS and AndroidShared: Syntactic Text Processing・Text Segmentation