Multimodal
15.1 Vision-Language Models
| Model | Stars | Description |
|---|---|---|
| LLaVA-JP | 64 | Japanese VLM trained using the LLaVA method |
| BLIP2-Japanese | 14 | Japanese version of the BLIP2 Q-former |
| clip-japanese | 13 | Japanese CLIP model |
| llm-jp-vila | 10 | VILA-based VLM training |
| llm-jp-eval-mm | 44 | Multimodal LLM evaluation |
LLaVA-JP is a Vision-Language Model that applies the LLaVA method to Japanese, with Stars: 64 — the most attention among Japanese VLMs.
heron-vlm-leaderboard (Stars: 6): A VLM benchmark tool using two datasets: LLaVA-Bench-In-the-Wild and the Japanese HERON Bench.
Multimodal Models on Hugging Face
| Model | HF DL | Likes | Category | Description |
|---|---|---|---|---|
| manga-ocr-base (kha-white) | 299k | 192 | image-to-text | Manga OCR; handles vertical/horizontal text |
| meiki.txt.recognition.v0 (rtr46) | 42k | 7 | image-to-text | Game text recognition (D-FINE/MobileNetV4) |
| meiki.text.detect.v0 (rtr46) | 22k | 3 | image-to-text | Text detection for video games |
| sarashina2.2-ocr (sbintuitions) | 1k | 34 | image-to-text | 3B; Markdown/HTML/LaTeX output |
| sarashina2.2-vision-3b (sbintuitions) | 1k | 20 | image-to-text | 3B VLM; SigLIP encoder |
manga-ocr-base has very high popularity on Hugging Face with 299k/week downloads and 192 Likes. Using a Vision Encoder-Decoder architecture, it reads Japanese manga text (including furigana) in both vertical and horizontal writing with high accuracy.
The meikiocr models (rtr46) are D-FINE-based MobileNetV4 text detection and recognition models for Japanese video games. At 45k/week for recognition and 29k/week for detection, they show that practical use of Japanese image-text recognition is spreading beyond manga.
sarashina2.2-ocr (SB Intuitions) is a 3B-parameter end-to-end OCR model that parses Japanese and English documents into Markdown, with tables as HTML and formulas as LaTeX. It integrates a SigLIP2-based vision encoder with Sarashina2.2-3B-Instruct.
VLM Evaluation and Datasets
- eval_vertical_ja (llm-jp): Benchmark for evaluating multimodal LLM performance on vertically written Japanese text.
- modelvista-3lang (kuramitsulab): VLM evaluation benchmark for software diagram understanding (Japanese, English, Korean). Stars: 2.
- business-slide-questions (Stockmark): VQA benchmark "BusinessSlideVQA" for business slides. Stars: 2.
- YouCook2-JP (nlab-mpg): Japanese translation of the YouCook2 cooking video caption dataset.
15.2 Image Captioning
STAIR-captions (STAIR-Lab-CIT/STAIR-captions): A large-scale Japanese image captioning dataset. Stars: 90. Provides Japanese captions for MS-COCO images.
ClipCap-for-Japanese (Stars: 13), SAT-for-Japanese (Stars: 2): Japanese implementations of image captioning methods.
JaSPICE (Stars: 9): An image captioning evaluation metric. An automated evaluation method for Japanese using predicate-argument structure.