15

Multimodal

15.1 Vision-Language Models

Model Stars Description
LLaVA-JP 64 Japanese VLM trained using the LLaVA method
BLIP2-Japanese 14 Japanese version of the BLIP2 Q-former
clip-japanese 13 Japanese CLIP model
llm-jp-vila 10 VILA-based VLM training
llm-jp-eval-mm 44 Multimodal LLM evaluation

LLaVA-JP is a Vision-Language Model that applies the LLaVA method to Japanese, with Stars: 64 — the most attention among Japanese VLMs.

heron-vlm-leaderboard (Stars: 6): A VLM benchmark tool using two datasets: LLaVA-Bench-In-the-Wild and the Japanese HERON Bench.

LLaVA-JP
Japanese VLM trained using the LLaVA method
64
BLIP2-Japanese
Japanese version of the BLIP2 Q-former
14
clip-japanese
Japanese CLIP model
13
llm-jp-vila
VILA-based VLM training
10
llm-jp-eval-mm
Multimodal LLM evaluation
44
heron-vlm-leaderboard
heron-vlm-leaderboard (Stars: 6): A VLM benchmark tool using two datasets: LLaVA-Bench-In-the-Wild and the Japanese HERON Bench.
6

Multimodal Models on Hugging Face

Model HF DL Likes Category Description
manga-ocr-base (kha-white) 299k 192 image-to-text Manga OCR; handles vertical/horizontal text
meiki.txt.recognition.v0 (rtr46) 42k 7 image-to-text Game text recognition (D-FINE/MobileNetV4)
meiki.text.detect.v0 (rtr46) 22k 3 image-to-text Text detection for video games
sarashina2.2-ocr (sbintuitions) 1k 34 image-to-text 3B; Markdown/HTML/LaTeX output
sarashina2.2-vision-3b (sbintuitions) 1k 20 image-to-text 3B VLM; SigLIP encoder

manga-ocr-base has very high popularity on Hugging Face with 299k/week downloads and 192 Likes. Using a Vision Encoder-Decoder architecture, it reads Japanese manga text (including furigana) in both vertical and horizontal writing with high accuracy.

The meikiocr models (rtr46) are D-FINE-based MobileNetV4 text detection and recognition models for Japanese video games. At 45k/week for recognition and 29k/week for detection, they show that practical use of Japanese image-text recognition is spreading beyond manga.

sarashina2.2-ocr (SB Intuitions) is a 3B-parameter end-to-end OCR model that parses Japanese and English documents into Markdown, with tables as HTML and formulas as LaTeX. It integrates a SigLIP2-based vision encoder with Sarashina2.2-3B-Instruct.

manga-ocr-base (kha-white)
Manga OCR; handles vertical/horizontal text
299k 192
meiki.txt.recognition.v0 (rtr46)
Game text recognition (D-FINE/MobileNetV4)
42k 7
meiki.text.detect.v0 (rtr46)
Text detection for video games
22k 3
sarashina2.2-ocr (sbintuitions)
3B; Markdown/HTML/LaTeX output
1k 34
sarashina2.2-vision-3b (sbintuitions)
3B VLM; SigLIP encoder
1k 20

VLM Evaluation and Datasets

  • eval_vertical_ja (llm-jp): Benchmark for evaluating multimodal LLM performance on vertically written Japanese text.
  • modelvista-3lang (kuramitsulab): VLM evaluation benchmark for software diagram understanding (Japanese, English, Korean). Stars: 2.
  • business-slide-questions (Stockmark): VQA benchmark "BusinessSlideVQA" for business slides. Stars: 2.
  • YouCook2-JP (nlab-mpg): Japanese translation of the YouCook2 cooking video caption dataset.
eval_vertical_ja
- eval_vertical_ja (llm-jp): Benchmark for evaluating multimodal LLM performance on vertically written Japanese text.
modelvista-3lang
- modelvista-3lang (kuramitsulab): VLM evaluation benchmark for software diagram understanding (Japanese, English, Korean). Stars: 2.
2
business-slide-questions
- business-slide-questions (Stockmark): VQA benchmark "BusinessSlideVQA" for business slides. Stars: 2.
2
YouCook2-JP
- YouCook2-JP (nlab-mpg): Japanese translation of the YouCook2 cooking video caption dataset.

15.2 Image Captioning

STAIR-captions (STAIR-Lab-CIT/STAIR-captions): A large-scale Japanese image captioning dataset. Stars: 90. Provides Japanese captions for MS-COCO images.

ClipCap-for-Japanese (Stars: 13), SAT-for-Japanese (Stars: 2): Japanese implementations of image captioning methods.

JaSPICE (Stars: 9): An image captioning evaluation metric. An automated evaluation method for Japanese using predicate-argument structure.


STAIR-captions
STAIR-captions (STAIR-Lab-CIT/STAIR-captions): A large-scale Japanese image captioning dataset. Stars: 90. Provides Japanese captions for MS-COCO images.
90
SAT-for-Japanese
ClipCap-for-Japanese (Stars: 13), SAT-for-Japanese (Stars: 2): Japanese implementations of image captioning methods.
13
JaSPICE
JaSPICE (Stars: 9): An image captioning evaluation metric. An automated evaluation method for Japanese using predicate-argument structure.
9