datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
dclm-baseline-1.0
DCLM-baseline
DCLM-baseline is a 4T token / 3B document pretraining dataset that achieves strong performance on language model benchmarks.
Below are comparisions of model trained on DCLM-baseline with other models in the 7B regime.
Model
Params
Tokens
Open dataset?
CORE
MMLU
EXTENDED
Open weights, closed datasets
Llama2
7B
2T
✗
49.2
45.8
34.1
DeepSeek
7B
2T
✗
50.7
48.5
35.3
Mistral-0.3
7B
?
✗
57.0
62.7
45.1
QWEN-2
7B
?
✗
57.5
71.9
50.5
Llama3
8B
15T
✗… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0.GUI_BASED_PLATFORMlm-eval-results-princeton-nlp-Llama-3-Base-8B-SFT-RDPO-private
Dataset Card for Evaluation run of princeton-nlp/Llama-3-Base-8B-SFT-RDPO
Dataset automatically created during the evaluation run of model princeton-nlp/Llama-3-Base-8B-SFT-RDPO
The dataset is composed of 62 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 7 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the… See the full description on the dataset page: https://huggingface.co/datasets/nyu-dice-lab/lm-eval-results-princeton-nlp-Llama-3-Base-8B-SFT-RDPO-private.AToMiC-Baselines
AToMiC Prebuilt Indexes
Example Usage:
Reproduction
Toolkits:
https://github.com/TREC-AToMiC/AToMiC/tree/main/examples/dense_retriever_baselines
# Skip the encode and index steps, search with the prebuilt indexes and topics directly
python search.py \
--topics topics/openai.clip-vit-base-patch32.text.validation \
--index indexes/openai.clip-vit-base-patch32.image.faiss.flat \
--hits 1000 \
--output… See the full description on the dataset page: https://huggingface.co/datasets/TREC-AToMiC/AToMiC-Baselines.Unite-Base-Retrieval-Train
Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval
Statistics
Accessing Images and Videos
2025-06-19: We've updated the compressed archives for all image and video files to enable faster extraction.If you've already downloaded the previous files, there's no need to redownload them — the content remains exactly the same. The only difference lies in the compression method, which now allows for quicker… See the full description on the dataset page: https://huggingface.co/datasets/friedrichor/Unite-Base-Retrieval-Train.livesqlbench-base-lite-sqlite
🚀 LiveSQLBench-Base-Lite
A dynamic, contamination‑free benchmark for evaluating LLMs on complex, real‑world text‑to‑SQL tasks.
🌐 LiveSQLBench Website • 🌐 BIRD-INTERACT Project Page • 📄 Paper • 💻 LiveSQLBench GitHub • 💻 BIRD-INTERACT GitHub
Maintained by the 🦜 BIRD Team @ HKU & ☁️ Google Cloud
📊 LiveSQLBench Overview
LiveSQLBench (BIRD-SQL Pro v0.5) is a contamination-free, continuously evolving benchmark designed to evaluate LLMs on complex, real-world… See the full description on the dataset page: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite-sqlite.BenchMAX_Rule-based
Dataset Sources
Paper: BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models
Link: https://huggingface.co/papers/2502.07346
Repository: https://github.com/CONE-MT/BenchMAX
Dataset Description
BenchMAX_Rule-based is a dataset of BenchMAX, sourcing from IFEval, which is a rule-based benchmark for evaluating the instruction following capabilities in multilingual scenarios.
We extend the original dataset to 16 non-English languages by first… See the full description on the dataset page: https://huggingface.co/datasets/LLaMAX/BenchMAX_Rule-based.livesqlbench-base-full-v1
🚀 LiveSQLBench-Base-Full-v1
A dynamic, contamination‑free benchmark for evaluating LLMs on complex, real‑world text‑to‑SQL tasks.
🌐 Website/Leaderboard • 📄 Paper (coming soon) • 💻 GitHub • 🗄️ LiveSQLBench-Base-Lite • 🗄️ LiveSQLBench-Large-v1 • 🗄️ Bird-Interact (ICLR 2026 Oral)
Maintained by the 🦜 BIRD Team @ HKU & ☁️ Google Cloud
📊 LiveSQLBench Overview
LiveSQLBench (BIRD-SQL Pro v0.5) is a contamination-free, continuously evolving benchmark designed to… See the full description on the dataset page: https://huggingface.co/datasets/birdsql/livesqlbench-base-full-v1.so100_base_env
SO-100 Base Environment (TsFile)
Apache TsFile version of shreyasgite/so100_base_env.
Overview
A LeRobot teleoperation dataset recorded on an SO-100 arm performing a Lego
pick-and-place task: "Grasp a lego block and put it in the bin." Each episode
captures the synchronized robot joint state and commanded action at every control
step.
Robot: so100 (6-DoF arm: shoulder pan/lift, elbow flex, wrist flex/roll, gripper).
Episodes: 252 (single train split).
Frames: 98… See the full description on the dataset page: https://huggingface.co/datasets/THULab/so100_base_env.reasoning-base-20k
Dataset Card for Reasoning Base 20k
Dataset Details
Dataset Description
This dataset is designed to train a reasoning model. That can think through complex problems before providing a response, similar to how a human would. The dataset includes a wide range of problems from various domains (science, coding, math, etc.), each with a detailed chain of thought (COT) and the correct answer. The goal is to enable the model to learn and refine its reasoning process… See the full description on the dataset page: https://huggingface.co/datasets/KingNish/reasoning-base-20k.gpt-oss120b-generated-perfectblendMME-RealWorld-Base64
MME-RealWorld Dataset
This dataset contains multiple JSON files split into chunks. It includes information such as questions, images encoded in base64, and other related metadata.
Usage
You can load the dataset using the datasets library:
from datasets import load_dataset
dataset = load_dataset('yifanzhang114/MME-RealWorld-Base64', data_dir='MME-RealWorld')
dataset = load_dataset('yifanzhang114/MME-RealWorld-Base64', data_dir='MME-RealWorld-CN')
## the image can be… See the full description on the dataset page: https://huggingface.co/datasets/yifanzhang114/MME-RealWorld-Base64.livesqlbench-base-lite
🚀 LiveSQLBench-Base-Lite
A dynamic, contamination‑free benchmark for evaluating LLMs on complex, real‑world text‑to‑SQL tasks.
🌐 Website • 📄 Paper (coming soon) • 💻 GitHub • 🗄️ LiveSQLBench-Base-Full-v1 • 🗄️ LiveSQLBench-Large-v1 • 🗄️ Bird-Interact (ICLR 2026 Oral)
Maintained by the 🦜 BIRD Team @ HKU & ☁️ Google Cloud
📊 LiveSQLBench Overview
LiveSQLBench (BIRD-SQL Pro v0.5) is a contamination-free, continuously evolving benchmark designed to evaluate LLMs on… See the full description on the dataset page: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite.BananaMind-Base-Bench-1.1
BananaMind Base Bench 1.1
BananaMind Base Bench 1.1 is an English text-completion benchmark for base causal language models. It contains 350 individually authored examples across seven categories and reports one fixed-scale Overall Elo score.
This is not an instruction-following benchmark. Models receive plain text followed by four possible continuations. The official runner selects the continuation with the highest mean conditional token log-probability. It does not use a chat… See the full description on the dataset page: https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1.sorrel-T-qwen3-8b-base-seed0-documentsfixed-n-rb-er-cost-marginrl-qwen3-1.7b-base-math12k-token-mean-fixed-q0p8-run2-rollouts
fixed_n_rb_er_cost_marginrl_Qwen3-1.7B-Base_math12k_token_mean_fixed_q0.8_run2 rollouts
This dataset contains one compressed JSONL shard for every completed training
step. The step and rollout_index columns uniquely locate a rollout within
this training run. Run metadata and per-step row counts are recorded in
rollout_manifest.json.
sorrel-T-qwen3-1.7b-base-seed0-documentsmagpie-qwen2.5-pro-1m-v0.1-Qwen3-235B-A22B-Instruct-2507-FP8-generatedfranka_pnp_big100_base
Franka PnP Big-100 — Base
100 shelf pick-and-place demonstrations with no clutter: one coke can on the floor of a large shelf, a terracotta dish on the desk to the left.
Collected in Isaac Lab (Isaac Lab Arena) with a Franka Panda and a cuRobo-planned scripted expert. One successful episode per scene; the can position is sampled uniformly over the shelf floor (see Sampling). This is one of a pair of datasets: franka_pnp_big100_base (no clutter) and franka_pnp_big100_distract… See the full description on the dataset page: https://huggingface.co/datasets/lithyeon/franka_pnp_big100_base.BenchMAX_Model-based
Dataset Sources
Paper: BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models
Link: https://huggingface.co/papers/2502.07346
Repository: https://github.com/CONE-MT/BenchMAX
Dataset Description
BenchMAX_Model-based is a dataset of BenchMAX, sourcing from m-ArenaHard, which evaluates the instruction following capability via model-based judgment.
We extend the original dataset to include languages that are not supported by m-ArenaHard through… See the full description on the dataset page: https://huggingface.co/datasets/LLaMAX/BenchMAX_Model-based.codex-assistant-rollouts
basedlsg/codex-assistant-rollouts
Real-world agentic interaction logs from Codex rollouts, documenting debugging and coding trajectories.
cantonese-chinese-parallel-corpus-baseThis is a dataset of Cantonese-Written Chinese Parallel Corpus, containing 130k+ pairs of Cantonese and Traditional Chinese parallel sentences.
sorrel-T-qwen3-14b-base-seed0-documentsPUBMED_title_abstracts_2020_baselinebase-eval-rollouts
base eval rollouts
Eval rollouts (32 samples/problem) for ReasoningRegisters/base (untrained base model).
Split folders named step{k}_{benchmark}[_variant]; JSONL per shard: prompt, generation, correctness.
deepagent
DeepAgent
Hard, Docker-verifiable software-engineering benchmarks from real merged PRs
DeepAgent ships real_pr Harbor hardness packs: live-mined multi-file pull requests, clone@SHA agent images, held-out verifier tests, and Docker dual-truth (solution reward = 1, null reward = 0). Primary product work runs through the deepagent CLI in the GitHub monorepo.
Surface
Ref
Role
HF stable pin
this dataset revision main
Current product on Hub (N=9)
HF automation… See the full description on the dataset page: https://huggingface.co/datasets/BaseIntelligence/deepagent.hh-harmless-base-qwen3-8b-margin-dpo-margin-logssilicon-based-girlfriend-v2-dataset
矽基女友 v2 · 繁中角色扮演合成語料
繁體中文（臺灣）角色扮演的合成對話語料，2,109 筆、38,460 輪、角色輪合計 1,350 萬字元。
訓練出來的模型見 RX5950XT/silicon-based-girlfriend-v2-GGUF。
⚠️ 全部是模型合成的資料，不是真人對話。 內容包含成人向角色扮演，不適合未成年人。
僅供研究用途。所有角色皆為虛構成年人。
內容
檔案
內容
sharegpt_dataset.json
2,109 筆多輪對話，ShareGPT 格式（id / system / conversations）
grpo_prompts.json
648 題 GRPO 用的提示，與 SFT 語料零重疊
holdout_ids.json
100 筆 holdout ID 清單，這些已從 SFT 訓練集排除，供驗收用
general_probes.json
64 題通用能力探針（5 類），用來檢測微調後有無退化… See the full description on the dataset page: https://huggingface.co/datasets/RX5950XT/silicon-based-girlfriend-v2-dataset.sorrel-T-mistral-small-24b-base-seed0-documentsgpt-oss120b-generated-magpie-1m-v0.1