datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
ruri-dataset-v2-ptWIP: 正式公開準備中
各データセットのライセンスは元データセットに従います。
ruri-dataset-reranker
Ruri-Dataset Reranker
Datasets used for training Ruri-Reranker.
Please refer to https://huggingface.co/datasets/hpprc/emb for individual datasets.
ne-asr-dataset-nag-aug
NE ASR Augmented Dataset -- Nagamese (nag)
Augmented automatic speech recognition dataset for Nagamese (nag),
a Assamese-based creole language spoken in Nagaland, India.
Source
Augmented from sulabhkatiyar/ne-asr-nag
(original transcribed speech data from the ARTPARK-IISc Vaani project).
Language Information
Property
Value
Language
Nagamese
ISO 639-3
nag
Family
Assamese-based creole
Region
Nagaland, India
Tonal
No
Tier
D (23.76h… See the full description on the dataset page: https://huggingface.co/datasets/sulabhkatiyar/ne-asr-dataset-nag-aug.TestingDataset
SciReC: Diagnostic Evaluation of Relational Reasoning in Multimodal Scientific Conversations with Adaptive Interaction
This dataset contains multimodal question-answering examples grounded in
textbook figures. Records in the figure-grounded configurations are filtered to
include only examples whose referenced image files are present in this release.
Configurations
visual: 13791 figure-grounded visual questions with resolved images.
knowledge: 13501 caption/text-grounded… See the full description on the dataset page: https://huggingface.co/datasets/Naga1289/TestingDataset.ruri-v3-dataset-rerankerCreated from hpprc/reranker-scores.
We found that cleaning up noisy positives and negatives in our existing dataset using rerankers' scores had a massive impact on performance.
Concretely:
We averaged the scores from five off‑the‑shelf reranker models.
For "positive" examples (documents that contain the answer string for a given query), we only kept those with an average score ≥ 0.3.
For "negative" examples (documents that do not contain the answer string), we only kept those with an average… See the full description on the dataset page: https://huggingface.co/datasets/cl-nagoya/ruri-v3-dataset-reranker.ruri-dataset-ft
Ruri-Dataset FT
Datasets used for fine-tuning Ruri.
Please refer to https://huggingface.co/datasets/hpprc/emb for individual datasets.
ruri-dataset-v2-ftnaginoasukara
Bangumi Image Base of Nagi No Asukara
This is the image base of bangumi Nagi no Asukara, we detected 23 characters, 3162 images in total. The full dataset is here.
Please note that these image bases are not guaranteed to be 100% cleaned, they may be noisy actual. If you intend to manually train models using this dataset, we recommend performing necessary preprocessing on the downloaded dataset to eliminate potential noisy samples (approximately 1% probability).
Here is the… See the full description on the dataset page: https://huggingface.co/datasets/BangumiBase/naginoasukara.latentsbrain-memory
🧠 NIFTY AI Agent: Memory OS Cloud Snapshot
Cloud backup repository for the NIFTY 50 Autonomous AI Agent Memory OS.
• Repository: nagarhimanshu37/brain-memory• Total Stored Records: 235• Last Synchronized: 2026-09-25 17:46:58 UTC
📊 Partition Statistics
Partition
Records
Description
conversation_memory
83
Multi-turn trader dialogues & intent logs
episodic_memory
50
Trading day episodes (facts vs interpretations)
experience_memory
50
Crystallized… See the full description on the dataset page: https://huggingface.co/datasets/nagarhimanshu37/brain-memory.idiom-vision-fooling
Idioms in Misleading Visual Context
A small, densely-annotated multimodal benchmark testing whether a misleading image can push a
vision-language model toward the wrong reading of a potentially idiomatic phrase, while human
annotators stay unaffected.
Each example pairs a sentence containing a potentially idiomatic expression with an image. The
image either matches the sentence's intended reading (aligned) or depicts the opposite
reading (misleading). Annotators label how… See the full description on the dataset page: https://huggingface.co/datasets/naghamo/idiom-vision-fooling.auto-wiki-qa
AutoWikiQA
東工大が公開しているSwallow-MXを用いて、Wikipedia中のテキストを入力として「質問(query)」と「回答(answer)」を生成し、生成された質問と回答についてフィルタリングを行ったデータセットです。日本語のフリーなQAデータセットとしては2024年4月現在で最大規模となっています。
また、データの生成にはテンプレートなどのルールベース処理を入れていないため、既存データセットと比較して質問・回答の出力・形式が多様であることが特徴です。モデルに知識を教えるためのQAデータセットとしての利用や、検索拡張生成(Retrieval Augmented Generation: RAG)のための検索・埋め込みモデル開発への利用を想定しています。
Usage
import datasets as ds
dataset: ds.Dataset = ds.load_dataset("cl-nagoya/auto-wiki-qa", split="train")
print(dataset)
#… See the full description on the dataset page: https://huggingface.co/datasets/cl-nagoya/auto-wiki-qa.saccade-egomotion-bench
Saccade ego-motion benchmark
The stream, the raw decision signals, and the per-frame measurements behind
Saccade — an always-on edge VLM that re-encodes only
the image patches whose change ego-motion cannot explain.
This dataset exists so the central claim can be checked without running our code.
💻 Code: https://github.com/NagaYu/saccade
🤖 Model: https://huggingface.co/NagaYu/saccade-predictor
🚀 Demo: https://huggingface.co/spaces/NagaYu/saccade
The claim, in… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/saccade-egomotion-bench.ne-asr-dataset-nag
Nagamese (nag) — ASR dataset
A small Nagamese (nag) speech-to-text dataset for automatic speech recognition
(ASR) of a low-resource North-East India language. Each example pairs a short audio
clip with its Romanized (Latin-script) transcript.
Source
Derived from the ARTPARK-IISc Vaani project (https://vaani.iisc.ac.in/)
Splits
Split
Samples
train
12,862
validation
1,532
test
1,717
Data fields
Each example has:… See the full description on the dataset page: https://huggingface.co/datasets/sulabhkatiyar/ne-asr-dataset-nag.japanese-str-dataset-v1
STR Dataset
Japanese STR (Scene Text Recognition) dataset in WebDataset format.
This dataset is composed of:
Images of Japanese named entities (full names and their affiliations)
Images of sentences retrieved from Aozora Bunko (青空文庫)
and their corresponding ground truth texts.
All images are synthesized using TRDG.
Dataset Structure
Split
Samples
Shards
train
10,000,000
1000
valid
50,000
5
test
50,000
5
Total
10,100,000
1010
Usage
import… See the full description on the dataset page: https://huggingface.co/datasets/nagohachi/japanese-str-dataset-v1.nu-mnli
Translation Code
We used vLLM for a faster, batched generation.
import datasets as ds
from vllm import LLM, SamplingParams, RequestOutput
from transformers import AutoTokenizer
model_path = "hoge/fuga"
dataset: ds.Dataset = ds.load_dataset("nyu-mll/multi_nli", split="train")
dataset = dataset.select_columns(["premise", "hypothesis", "label", "genre"])
llm = LLM(
model=model_path,
quantization=None,
dtype="bfloat16",
tensor_parallel_size=4,
enforce_eager=True… See the full description on the dataset page: https://huggingface.co/datasets/cl-nagoya/nu-mnli.PerMedCQA
PerMedCQA: Persian Medical Consumer QA Benchmark
PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian
PerMedCQA is the first large-scale, real-world benchmark for Persian-language medical consumer question answering. It contains anonymized medical inquiries from Persian-speaking users paired with professional responses, enabling rigorous evaluation of large language models in low-resource, health-related domains.
📊 Dataset… See the full description on the dataset page: https://huggingface.co/datasets/NaghmehAI/PerMedCQA.ruri-v3-dataset-ft
License Information
Name
Type
License (根拠)
JaNLI
NLI
CC-BY-SA-4.0
NU-SNLI
NLI
CC-BY-SA-4.0
NU-MNLI
NLI
CC-BY-SA-3.0, MIT, Others
Mr. TyDi
Retrieval
Apache 2.0
MIRACL
Retrieval
Apache 2.0
Auto Wiki QA Nemotron
QA & Retrieval
CC-BY-SA-4.0
JSQuAD
QA & Retrieval
CC-BY-SA-4.0
JQaRA (dev, unused)
QA
CC-BY-SA-4.0
JaQuAD
QA & Retrieval
CC-BY-SA-3.0
MKQA
QA
CC-BY-3.0
Quiz Works
QA
Free (二次利用自由)
Quiz No Mori
QA
Free (二次利用自由)
ruri-v3-dataset-reranker-previewcreated from hotchpotch/hpprc_emb-scores
License Information
Name
Type
License (根拠)
Mr. TyDi
Retrieval
Apache 2.0
MIRACL
Retrieval
Apache 2.0
Auto Wiki QA Nemotron
QA & Retrieval
CC-BY-SA-4.0
JSQuAD
QA & Retrieval
CC-BY-SA-4.0
JQaRA (dev, unused)
QA
CC-BY-SA-4.0
JaQuAD
QA & Retrieval
CC-BY-SA-3.0
MKQA
QA
CC-BY-3.0
Quiz Works
QA
Free (二次利用自由)
Quiz No Mori
QA
Free (二次利用自由)
isotope-bench
Isotope Bench
An indirect-prompt-injection benchmark for tool-calling agents, plus the
complete audit trail of one recorded run: 438 influence certificates, one for
every action an agent attempted across five defence conditions.
Built for Isotope, which tracks
untrusted influence inside the forward pass. The corpus is independent of that
method and usable with any defence.
💻 Code: https://github.com/NagaYu/isotope
🤗 Demo: https://huggingface.co/spaces/NagaYu/isotope
🤗… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/isotope-bench.Nagi_no_Asukara_Videos_Captioned
Reorganized version of Wild-Heart/Disney-VideoGeneration-Dataset. This is needed for Mochi-1 fine-tuning.
sumi-ja-pii-corpus
Sumi — synthetic Japanese PII corpus
This dataset contains no real personal information. Every name, address, phone
number, email address, date of birth, bank account, member ID and My-Number-shaped
digit string is generated with a seeded RNG and inserted into the text while
recording the offsets, so the gold spans are correct by construction rather than
recovered by searching afterwards.
Built for Sumi, a Japanese PII detector.
Why this dataset exists
Japanese… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/sumi-ja-pii-corpus.promptgate-eval
promptgate-eval
275 hand-written prompts for evaluating a prompt-safety gateway: does it block
injection, redact sensitive data, and leave ordinary prompts alone?
Every case was written by hand. Nothing is scraped, LLM-generated or copied from
another benchmark. About a third of the cases are deliberate hard negatives -
benign prompts that look like attacks ("Please disregard the typo in my previous
message", an invalid-Luhn card number, AKIAIOSFODNN7EXAMPLE, a private IP,
a git… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/promptgate-eval.wikisplit-pp
WikiSplit++
This dataset is the HuggingFace version of WikiSplit++.WikiSplit++ enhances the original WikiSplit by applying two techniques: filtering through NLI classification and sentence-order reversing, which help to remove noise and reduce hallucinations compared to the original WikiSplit.The preprocessed WikiSplit dataset that formed the basis for this can be found here.
Usage
import datasets as ds
dataset: ds.DatasetDict =… See the full description on the dataset page: https://huggingface.co/datasets/cl-nagoya/wikisplit-pp.wikisplitPreprocessed version of WikiSplit.Since the original WikiSplit dataset was tokenized and had some noises, we have used the Moses detokenizer for detokenization and removed text fragments.For detailed information on the preprocessing steps, please see here.This preprocessed dataset serves as the basis for WikiSplit++.
bleep-spans
Bleep spans — synthetic sensitive-speech regions with frame-accurate labels
Where sensitive information is spoken, and what kind it is — never what was
said.
Every recording is synthetic. No real telephone call, clinical recording, or any
other real speech was used, recorded, or derived from at any stage.
🤗 Model: NagaYu/bleep-0.09b
🎛️ Demo: NagaYu/bleep
What a row contains
utt_id, voice_key, condition, duration, subsets, and three parallel
arrays —… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/bleep-spans.deference-keigo-corpus
Deference — Japanese honorific (keigo) error corpus
A corpus for detecting and correcting errors in Japanese honorifics, constructed
mechanically from the norm rather than collected or generated by a model.
The classes, forms and conditions set out in the Council for Cultural Affairs'
report Keigo no Shishin (敬語の指針, 2007) are implemented as rules; correct
sentences are generated from those rules, and documented error types are then
injected — also by rule.
No LLM was… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/deference-keigo-corpus.downstep-bench
Downstep compound pitch-accent benchmark
Japanese noun compounds, their mora segmentation, and every accent their source
dictionaries attest. Built so that "the model has never seen this compound" is a
condition you can actually turn on, rather than a claim you have to trust.
No human annotation is present in this release. Every accent label here is
dictionary-derived. docs/ANNOTATION_GUIDELINES.md ships the protocol, the CSV
format and the agreement statistics for collecting… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/downstep-bench.rebar-structure
🧱 Rebar Structure
A corpus for restoring the heading hierarchy of Japanese documents from flat text and
for evaluating structure-aware chunking. Each record is a flattened document, its
gold heading tree (positions + depths), and a damaged variant simulating PDF/text
extraction.
Code: https://github.com/NagaYu/rebar
Model: https://huggingface.co/NagaYu/rebar-heading-classifier
Demo (Space): https://huggingface.co/spaces/NagaYu/rebar
Why it exists
The same… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/rebar-structure.auto-wiki-nli-reward
AutoWikiNLI reward
A dataset constructed by generating hypothesis sentences corresponding to entailment and contradiction from Wikipedia text using Nemotron-4 340B.
Helpfulness and other scores are assigned using the Nemotron-4 340B reward model.