JMTEB is a Japanese text‑embedding benchmark featuring 5 tasks (clustering, classification, STS, retrieval, reranking) and 28 datasets, offering a one‑line evaluation script and inviting community contributions.
Description in other languages
Statistics
Statistics as of Nov 16, 2026
Hugging Face downloads (last 30 days)
3,000
Hugging Face likes
20
Listed
May 13, 2024
Downloads (30-day window) over time
Show as a table
| Month | DL (30d) |
|---|---|
| Dec 2025 | 6,000 |
| Jan 2026 | 3,000 |
| Feb 2026 | 1,000 |
| Mar 2026 | 2,000 |
| Apr 2026 | 3,000 |
| May 2026 | 4,000 |
| Jun 2026 | 2,000 |
| Jul 2026 | 1,000 |
| Aug 2026 | 3,000 |
| Sep 2026 | 2,000 |
| Oct 2026 | 3,000 |
| Nov 2026 | 3,000 |
Downloads over the 30 days before the last snapshot of each month.
Popular in the same category
- wikipedia-passages-jawiki-embeddingshotchpotch731Japanese Wikipedia sentences are transformed into various embeddings and a FAISS index, offering a Hugging Face Space demo, conversion scripts, and evaluations of search, Q&A, and OpenAI text‑embedding‑3‑small for RAG; embeddings are OpenAI‑licensed, others CC‑BY‑SA‑4.0.
- JQaRAhotchpotch436A Japanese QA dataset for evaluating Retrieval‑Augmented Generation (RAG), built from JAQKET questions and Wikipedia passages with gold retrieval‑relevance labels, released on HuggingFace and GitHub and scored primarily by nDCG@10.
- msmarco-ja-hard-negativeshotchpotch408Japanese hard‑negative mining and evaluation on the MSMARCO‑JA dataset, detailing normalization, filtering, BAAI/BGE reranker selection, random sampling, SPLADE‑based ranking, and chi‑square analysis showing statistically significant higher positive rates than the original mMARCO(ja).
- RAG-Evaluation-Dataset-JAallganize246Provides a Japanese RAG benchmark across five industry domains—finance, telecom, manufacturing, public, retail—by publishing datasets, an automatic evaluation framework, and comparative results for models such as Claude 3.5‑Sonnet, GPT‑4o, and others.
- Japanese-RAG-Generator-Benchmarkneoai-inc113Japanese RAG Generator Benchmark (J‑RAGBench) supplies a multi‑category QA dataset—covering Integration, Reasoning, Logical, Table, and Abstention—designed to evaluate Japanese RAG generators, built with human effort and GPT‑4.1, and released under CC BY‑SA 4.0.
- emilia-yodasTTS-AGI18kDataset of dialogues and lore from the Fate/Stay Night character “Emilia”, formatted for training and evaluating conversational language models.
Resources in similar research fields
ja.text8Hironsan113Japanese text8 corpus for word embedding.GitHubShared: Representation Learning・Semantic Text Processing
JapaneseEmbeddingEvaloshizo184Japanese Embedding EvaluationGitHubShared: Representation Learning・Semantic Text Processing
evaluate_japanese_w2vshihono12script to evaluate pre-trained Japanese word2vec model on Japanese similarity datasetGitHubShared: Responsible & Trustworthy NLP・Representation Learning
anlp_nlp2021_d3-1arusl1This repository contains codes related to the experiments in "An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification"GitHubShared: Information Extraction & Text Mining・Responsible & Trustworthy NLP- plamo-embedding-1bpfnet486kPLaMo‑Embedding‑1B is a Japanese text‑embedding model from Preferred Networks that converts Japanese text into vectors for information retrieval, classification, and clustering, shows strong performance on the JMTEB benchmark, and is freely available under an Apache v2.0 license.HF modelShared: Representation Learning・Semantic Text Processing
mtebembeddings-benchmark3.4kMTEB: State-of-the-art evaluation of embeddings across languages and modalities.MultilingualShared: Embeddings, search & RAG・Corpora & datasets