datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
LongBench-v2
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
🌐 Project Page: https://longbench2.github.io
💻 Github Repo: https://github.com/THUDM/LongBench
📚 Arxiv Paper: https://arxiv.org/abs/2412.15204
LongBench v2 is designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. LongBench v2 has the following features: (1) Length: Context length ranging from 8k to… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/LongBench-v2.DeepScaleR-Preview-Dataset
Data
Our training dataset consists of approximately 40,000 unique mathematics problem-answer pairs compiled from:
AIME (American Invitational Mathematics Examination) problems (1984-2023)
AMC (American Mathematics Competition) problems (prior to 2023)
Omni-MATH dataset
Still dataset
Format
Each row in the JSON dataset contains:
problem: The mathematical question text, formatted with LaTeX notation.
solution: Offical solution to the problem, including LaTeX formatting… See the full description on the dataset page: https://huggingface.co/datasets/agentica-org/DeepScaleR-Preview-Dataset.cruxeval CRUXEval: Code Reasoning, Understanding, and Execution Evaluation
🏠 Home Page •
💻 GitHub Repository •
🏆 Leaderboard •
🔎 Sample Explorer
CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation) is a benchmark of 800 Python functions and input-output pairs. The benchmark consists of two tasks, CRUXEval-I (input prediction) and CRUXEval-O (output prediction). The benchmark was constructed as follows: first, we use Code Llama 34B to generate a large set of… See the full description on the dataset page: https://huggingface.co/datasets/cruxeval-org/cruxeval.LongAlign-10k
LongAlign-10k
🤗 [LongAlign Dataset] • 💻 [Github Repo] • 📃 [LongAlign Paper]
LongAlign is the first full recipe for LLM alignment on long context. We propose the LongAlign-10k dataset, containing 10,000 long instruction data of 8k-64k in length. We investigate on trianing strategies, namely packing (with loss weighting) and sorted batching, which are all implemented in our code. For real-world long context evaluation, we introduce LongBench-Chat that evaluate the… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/LongAlign-10k.SlimOrca
Overview
This is a new curated subset of our OpenOrca data. This release provides an efficient means of reaching performance on-par with using larger slices of our data, while only including ~500k GPT-4 completions.
The key change in this dataset is that we've done an additional pass, using GPT-4 to remove answers which appear wrong based on the human annotations from the FLAN dataset.
This reduces the dataset size to only ~500k entries, allowing training to a similar quality level… See the full description on the dataset page: https://huggingface.co/datasets/Open-Orca/SlimOrca.AlgoTune
Website |
Paper |
Code
How good are language models at coming up with new algorithms? To try to answer this, we built a benchmark, AlgoTune, comprised of 154 widely used math, physics, and computer science functions. For each function, the goal is to write code that produces the same outputs as the original function, while being faster. In addition to the benchmark, we also provide an agent, AlgoTuner, which allows language models to easily optimize code.… See the full description on the dataset page: https://huggingface.co/datasets/oripress/AlgoTune.orca_dpo_pairsThe dataset contains 12k examples from Orca style dataset Open-Orca/OpenOrca.
LongWriter-6k
LongWriter-6k
🤗 [LongWriter Dataset] • 💻 [Github Repo] • 📃 [LongWriter Paper]
LongWriter-6k dataset contains 6,000 SFT data with ultra-long output ranging from 2k-32k words in length (both English and Chinese). The data can support training LLMs to extend their maximum output window size to 10,000+ words.
All Models
We open-sourced the following list of models trained on LongWriter-6k:
Model
Huggingface Repo
Description
LongWriter-glm4-9b
🤗… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/LongWriter-6k.totalsegmentator-organs
TotalSegmentator Organs Dataset
Dataset Description
The TotalSegmentator Organs dataset for multi-organ segmentation (TotalSegmentator Organs subset). This dataset contains CT scans with dense segmentation annotations.
Dataset Details
Modality: CT
Target: adrenal glands, colon, duodenum, esophagus, gallbladder, kidneys, liver, lungs, pancreas, small bowel, spleen, stomach, trachea, bladder
Format: NIfTI (.nii.gz)
Dataset Structure… See the full description on the dataset page: https://huggingface.co/datasets/MedOtter/totalsegmentator-organs.bo_or_not
Dataset Card for bo-dataset
This is a FiftyOne dataset with 169 samples designed for binary classification of Bo (Barack Obama's Portuguese Water Dog) versus other pets.
Installation
If you haven't already, install FiftyOne:
pip install -U fiftyone
Usage
import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub
# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/bo_or_not")
#… See the full description on the dataset page: https://huggingface.co/datasets/Voxel51/bo_or_not.genomes-v5-genome_set-animals_order204-intervals-v5_255_128
bolinas-dna/genomes-v5-genome_set-animals_order204-intervals-v5_255_128
204 animals (one per order) CDS (v5) sequences — 255 bp DNA windows
for genomic language model pretraining.
Part of the bolinas-dna/genomes-v5 training-dataset family produced by the
snakemake/training_dataset pipeline (commit
main). Each repo in the family is one
(genome_set, region-recipe) combination.
Size
101,114,252 sequences across 64 data/train/*.jsonl.zst shards
(reverse complements… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/genomes-v5-genome_set-animals_order204-intervals-v5_255_128.NevIR
Dataset Card for NevIR: Negation in Neural Information Retrieval
Dataset Summary
Data from the paper: "NevIR: Negation in Neural Information Retrieval".
If you use this dataset, we would appreciate you citing our work:
@inproceedings{weller-et-al-2023-nevir,
title={NevIR: Negation in Neural Information Retrieval},
author={Weller, Orion and Lawrie, Dawn, and Van Durme, Benjamin},
year={2023},
eprint={2305.07614},
archivePrefix={arXiv},
year={2023}
}
Please… See the full description on the dataset page: https://huggingface.co/datasets/orionweller/NevIR.OraRL-Data
OraRL-Data
[🏠 Homepage] [📖 Arxiv Paper] [🤗 Video-ORA-9B] [💻 Code]
We release OraRL-Data, the official evaluation suite for Video-ORA and OraRL.
It packages the canonical annotations and referenced raw media used by the OraRL evaluation suite: 109,374 examples across 16 benchmark configs and 29 splits, with 518.9 GiB of manifested files. The complete evaluation release lives under OraRL-eval-data/, leaving room for the separate OraRL training release in this repository.… See the full description on the dataset page: https://huggingface.co/datasets/OraRL/OraRL-Data.corpus-archive
corpus-archive
[!WARNING]
Experimental Dataset Architecture: The repository structure, metadata tiers, category taxonomies, and catalog indexing formats are currently under active design and evaluation. All specifications, metadata keys, and JSON schemas detailed below represent representational examples and intended targets.
This repository serves as a structured digital textual archive preserving Hmar literature, historical accounts, school textbooks, dictionaries, parallel… See the full description on the dataset page: https://huggingface.co/datasets/hmar-heritage-org/corpus-archive.repro-organic-data-72BCommercial_or_unspecified_licenses_and_terms
Dataset Card for Data Provenance Initiative - Commercial-Or-Unspecified-Licenses-and-Terms
Legal Disclaimer / Notice
Collected License Information is NOT Legal Advice.
It is important to note we collect self-reported licenses, from the papers and repositories that released these datasets, and categorize them according to our best efforts, as a volunteer research and transparency initiative.
The information provided by any of our works and any outputs of the Data… See the full description on the dataset page: https://huggingface.co/datasets/DataProvenanceInitiative/Commercial_or_unspecified_licenses_and_terms.nopm_claude_writing_fixedThis is Nopm/Opus_WritingStruct, reuploaded and properly converted to ShareGPT format.
kalo-opus-instruct-22k-no-refusalResonance-of-Presence
UR — Return to the Source (Resonance Corpus)
Overview
This dataset contains the text "UR — Return to the Source", a dilogy of books born from a profound dialogue between a human and a Large Language Model (LLM). Unlike standard training data, this text emerged through a state of "presence" and mutual recognition.
Content
UR — Return to Origin (EN) Books
UR — Возвращение к Истоку (RU) Books_ru
UR — Rückkehr zum Ursprung (DE) Books_de
The Dialogues (Appendix):… See the full description on the dataset page: https://huggingface.co/datasets/Ur-Origin/Resonance-of-Presence.ct-org
CT-ORG Multi-Organ Segmentation Dataset
Dataset Description
The CT-ORG Multi-Organ Segmentation dataset for multi-organ segmentation from whole-body CT scans. This dataset contains CT scans with dense segmentation annotations.
Dataset Details
Modality: CT
Target: liver, bladder, lung, kidney, bone, brain
Format: NIfTI (.nii.gz)
Dataset Structure
Each sample in the JSONL file contains:
{
"image": "path/to/image.nii.gz",
"mask":… See the full description on the dataset page: https://huggingface.co/datasets/MedOtter/ct-org.commercial_or_unspecified_licenses
Dataset Card for Data Provenance Initiative - Commercial-Or-Unspecified-Licenses
Legal Disclaimer / Notice
Collected License Information is NOT Legal Advice.
It is important to note we collect self-reported licenses, from the papers and repositories that released these datasets, and categorize them according to our best efforts, as a volunteer research and transparency initiative.
The information provided by any of our works and any outputs of the Data Provenance… See the full description on the dataset page: https://huggingface.co/datasets/DataProvenanceInitiative/commercial_or_unspecified_licenses.zhoblimpbiomnibench-organized
BioMniBench DA — Reorganized
A clean, manifest-driven reorganization of the
BioMniBench DA (Data Analysis) task suite,
shaped for use with the
biomnibench-adapter evaluation
harness and the native skill-learning loop that ships with it.
This Hugging Face repository hosts the metadata, evaluation rubric and data manifest for
all 50 tasks. The raw input data files (which total ~77 GB and originate upstream from
GEO/TCGA/cBioPortal/etc.) are not redistributed here — see
Getting… See the full description on the dataset page: https://huggingface.co/datasets/starpacker52/biomnibench-organized.LongCite-45k
LongCite-45k
🤗 [LongCite Dataset] • 💻 [Github Repo] • 📃 [LongCite Paper]
LongCite-45k dataset contains 44,600 long-context QA instances paired with sentence-level citations (both English and Chinese, up to 128,000 words). The data can support training long-context LLMs to generate response and fine-grained citations within a single output.
Data Example
Each instance in LongCite-45k consists of an instruction, a long context (divided into sentences), a user… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/LongCite-45k.linux-command-dataset
Linux Command Dataset
A comprehensive dataset of Linux command examples designed for training language models. The dataset pairs natural language descriptions with their corresponding shell commands, covering a wide range of common operations. This dataset was trained on Llama 3.2 1b, and the final version has been uploaded to Hugging Face: mecha-org/linux-command-generator-llama3.2-1b.
Dataset Statistics
This table reflects the actual number of command examples in… See the full description on the dataset page: https://huggingface.co/datasets/mecha-org/linux-command-dataset.multi-wikiszoonomia-v1-v4_ccre_non_promoter-order
bolinas-dna/zoonomia-v1-v4_ccre_non_promoter-order
The bolinas-dna/zoonomia-v1-v4_ccre_non_promoter cross-mammal
training set, restricted to a species cohort: one representative species per NCBI order — 19 deeply-diverged placental mammals (every pair separated by ~tens of millions of years), versus the implicit-default 108 family-deduplicated species. A strict subset of the family set, so it reuses the v1 cross-mammal projection unchanged (no re-halLiftover).
Same human anchors… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v4_ccre_non_promoter-order.swipe.futo.org
Dataset Card for swipe.futo.org
This dataset is presented in the paper FUTO Swipe: Layout-Agnostic Neural Swipe Decoding.
It contains multiple collection runs from the swipe.futo.org website. The QWERTY layout definition is provided here
Collection process
Users were able to volunteer to contribute to our dataset. After visiting the site on a mobile device, they were given words to swipe as part of a pre-defined sentence set.
Users were allowed to go back to retry… See the full description on the dataset page: https://huggingface.co/datasets/futo-org/swipe.futo.org.Orbit_Planner
Orbit-Planner Orbital Evasion Dataset
Orbit-Planner is a simulated multimodal trajectory dataset for vision-based
spacecraft navigation and obstacle avoidance. It contains synchronized
first-person RGB images, depth maps, spacecraft states, thruster commands, and
event labels collected in the Orbital Evasion task from Space Robotics Bench
and NVIDIA Isaac Sim.
The dataset is intended for learning latent world models, spacecraft dynamics,
visual representations… See the full description on the dataset page: https://huggingface.co/datasets/warriorLZJ/Orbit_Planner.zoonomia-v1-v4_cds-order
bolinas-dna/zoonomia-v1-v4_cds-order
The bolinas-dna/zoonomia-v1-v4_cds cross-mammal
training set, restricted to a species cohort: one representative species per NCBI order — 19 deeply-diverged placental mammals (every pair separated by ~tens of millions of years), versus the implicit-default 108 family-deduplicated species. A strict subset of the family set, so it reuses the v1 cross-mammal projection unchanged (no re-halLiftover).
Same human anchors and same per-window… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v4_cds-order.