datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SWE-bench-extraNote: This dataset has an improved and significantly larger successor: SWE-rebench.
Dataset Summary
SWE-bench Extra is a dataset that can be used to train or evaluate agentic systems specializing in resolving GitHub issues. It is based on the methodology used to build SWE-bench benchmark and includes 6,415 Issue-Pull Request pairs sourced from 1,988 Python repositories.
Dataset Description
The SWE-bench Extra dataset supports the development of software engineering agents… See the full description on the dataset page: https://huggingface.co/datasets/nebius/SWE-bench-extra.commonsense-qaqascsiqatweet_sentiment_extraction
TweetSentimentExtractionClassification
An MTEB dataset
Massive Text Embedding Benchmark
Task category
t2c
Domains
Social, Written
Reference
https://www.kaggle.com/competitions/tweet-sentiment-extraction/overview
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["TweetSentimentExtractionClassification"])
evaluator = mteb.MTEB(task)
model =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/tweet_sentiment_extraction.Multitask-National-Speech-Corpus-v1-extendpiqasciqSWE-bench-extra-500starcoder2data-extras
StarCoder2 Extras
This is the dataset of extra sources (besides Stack v2 code data) used to train the StarCoder2 family of models. It contains the following subsets:
Kaggle (kaggle): Kaggle notebooks from Meta-Kaggle-Code dataset, converted to scripts and prefixed with information on the Kaggle datasets used in the notebook. The file headers have a similar format to Jupyter Structured but the code content is only one single script.
StackOverflow (stackoverflow): stackoverflow… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/starcoder2data-extras.hle-extractextracel_waveforms
Spike waveform shards (derived from IBL dandiset 000409)
A curated collection of per-spike multichannel waveform windows extracted from NWB assets in the DANDI dandiset 000409 (International Brain Laboratory, IBL). This repository contains the extractor and uploader used to produce Parquet shards; this README documents the derived dataset, its provenance, format, and how to reproduce it.
Short description
Parquet shards of spike waveform windows (channels × timesteps).… See the full description on the dataset page: https://huggingface.co/datasets/rokaijano/extracel_waveforms.logiqarelbench-v2-extra
RelBench v2 extras
Everything RelBench v2 added on top of v1: the v2-only
databases (with their tasks), and the v2-only tasks defined on the v1 databases. The v1
databases and tasks stay in
stanford-star/relbench.
Same self-describing manifest layout (plain parquet + manifest.yaml):
<dataset>/
manifest.yaml # tables, primary keys, foreign-key graph, val/test timestamps
schema.svg # ER diagram
db/*.parquet # relational… See the full description on the dataset page: https://huggingface.co/datasets/stanford-star/relbench-v2-extra.nemotron_extra_sft_parquetomni_extract_bench
Omni Extract Bench
We weren’t satisfied with the current benchmarking options for extraction. They were biased, didn’t use realistic data and were hard to audit. Our view is that an extraction benchmark should do two things:
Help customers choose the right vendor; and
Give engineers a way to diagnose what’s actually going wrong in a given model.
That’s why we built OmniExtractBench.
OmniExtractBench is a comprehensive structured extraction benchmark, developed by Datalab.… See the full description on the dataset page: https://huggingface.co/datasets/datalab-to/omni_extract_bench.riddles_extendedguru-RL-92k-extra-info-compressed
Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
Note for this extra-info-compressed data version!
The dataset provided in this repository is specifically intended for use with the latest release of VeRL (v0.4.0). Since VeRL rl_dataset.py processes datasets as datasets.Dataset, it is essential that the structure of all Parquet files remains fully consistent. This repository is designed to meet that requirement.
In this repo, the… See the full description on the dataset page: https://huggingface.co/datasets/IFM/guru-RL-92k-extra-info-compressed.external-benchmarking
Vector Search Benchmarks
This repo contains datasets for benchmarking vector search performance, to help Superlinked prioritize integration partners.
For performing actual benchmarking on this dataset, see the github repository README.
Overview
We reviewed a number of publicly available datasets and noted 3 core problems + here is how this dataset fixes them:
Problems of other vector search benchmarks
How this dataset solves it
Not enough metadata of… See the full description on the dataset page: https://huggingface.co/datasets/superlinked/external-benchmarking.nanobeir-multilingual-extended
NanoBEIR Multilingual Extended Dataset
This dataset extends the NanoBEIR multilingual collection with Japanese and Korean translations.
Dataset Structure
Each configuration follows the pattern <BASE>_<LANG> with splits:
corpus: Document corpus
queries: Search queries
qrels: Query relevance judgments (when available)
Languages
Arabic (ar), German (de), English (en), Spanish (es), French (fr)
Italian (it), Norwegian (no), Portuguese (pt), Swedish (sv)… See the full description on the dataset page: https://huggingface.co/datasets/LiquidAI/nanobeir-multilingual-extended.tool-output-extraction-swebench
Tool Output Extraction Dataset
Paper | Code
Training data for squeez — a small model that prunes verbose coding agent tool output to only the evidence the agent needs next.
Task
Task-conditioned context pruning of a single tool observation for coding agents.
Given a focused extraction query and one verbose tool output, return the smallest verbatim evidence block(s) the agent should read next.
The model copies lines from the tool output — it never rewrites, summarizes, or… See the full description on the dataset page: https://huggingface.co/datasets/KRLabsOrg/tool-output-extraction-swebench.realmirror-extra-datasets
Statement
This dataset is only used for data augmentation and not for training!
If you need to obtain benchmark train data for RealMirror, please refer to:
RealMirror train data
Citation
If you find this work helpful in your research, please consider giving this repo a star ⭐ and citing our paper:
@article{tai2025realmirror,
title={RealMirror: A Comprehensive, Open-Source Vision-Language-Action Platform for Embodied AI},
author={Tai, Cong and Zheng, Zhaoyu and Long… See the full description on the dataset page: https://huggingface.co/datasets/zte-terminators/realmirror-extra-datasets.mbpp_extended
Dataset Card for "mbpp_extended"
More Information needed
scrapinghub-article-extraction-benchmark
Scrapinghub Article Extraction Benchmark
This dataset was originally created and distributed under MIT License by Scrapinghub on GitHub: github.com/scrapinghub/article-extraction-benchmark
It is mirrored on the HuggingFace Hub as a convenience.
tallyqa_extracted_featuresELSA_D3_external_test
ELSA - Multimedia use case
from datasets import load_dataset
elsa_test_set = load_dataset("elsaEU/ELSA_D3", split="test_set", streaming=True)
elsa_test_set_transf = load_dataset("elsaEU/ELSA_D3", split="test_set_transf", streaming=True)
Official page of the D3 external test set, composed by two different splits.
ELSA Multimedia D3 is a large collection of Deep Fake images, generated using diffusion models
This dataset was developed as part of the EU project ELSA.… See the full description on the dataset page: https://huggingface.co/datasets/elsaEU/ELSA_D3_external_test.govdocs1-by-extension
govdocs1 Dataset: By File Extension
[!NOTE]
PDFs from govdocs1 are at this repo in "raw" file form - no simple "mostly correct" way to convert to text
Markdown-parsed versions of documents in govdocs1 with light filtering.
Usage
Load specific file formats (e.g., .doc files) parsed to markdown with pandoc:
from datasets import load_dataset
# Replace "doc" with desired config name
dataset = load_dataset("BEE-spoke-data/govdocs1-by-extension", "doc")… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-by-extension.PowerBI_ExtractsDATA SOURCES
GitHub
Microsoft Fabric Community
CREDITS
Primary Parser: Hugoberry's PBIXRay
Manual Parser (Troublesome Files): Didier Terrien's PowerBI SideTools
CSV Extractor (Troublesome Files): Bravo by SQLBI
Parquet Viewer (Check Parquet Outputs): Sal's ParquetViewer
573+574 removed due to pickle error.
bimanual_towel_folding-extendprice-tag-extraction
Price tag extraction dataset
This dataset contains images of price tags extracted from the Open Prices dataset, along with information extracted from this dataset.
It is intended to be used to train large visual language models (LVLMs) to extract information from price tag images, as part of the Open Prices project.
For more information about the format of this dataset, please refer to the documentation on llm-image-extraction datasets.
Dataset creation
A detailed… See the full description on the dataset page: https://huggingface.co/datasets/openfoodfacts/price-tag-extraction.