datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Llama-3-SynE-Dataset
📄 Report | 💻 GitHub Repo
🔍 English | 简体中文
Here is the continual pre-training dataset. The Llama-3-SynE model is available here.
News
🌟🌟 2024/12/17: We released the code used for continual pre-training and data preparation. The code contains detailed documentation comments.
✨✨ 2024/08/12: We released the continual pre-training dataset.
✨✨ 2024/08/10: We released the Llama-3-SynE model.
✨ 2024/07/26: We released the technical report, welcome to check it… See the full description on the dataset page: https://huggingface.co/datasets/RUC-AIBOX/Llama-3-SynE-Dataset.dolma-v1_7-305B-tokenized-llama3-nanosetTokenized (Llama 3) verison of NousResearch/dolma-v1_7-305B as a Nanotron dataset split into 10 GB chunks.
To download:
huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-305B-tokenized-llama3-nanoset --local-dir-use-symlinks False NousResearch/dolma-v1_7-305B-tokenized-llama3-nanoset
To recombine:
cat dolma-v1_7-305B-tokenized-llama3-nanoset/dolma-v1_7-305B-tokenized-llama3-nanoset.npy.* > dolma-v1_7-305B-tokenized-llama3-nanoset.npy
rm -rf… See the full description on the dataset page: https://huggingface.co/datasets/emozilla/dolma-v1_7-305B-tokenized-llama3-nanoset.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled
Every non-first chunk of every document carries a thought: the gpt-5.6-luna reasoning thought
where one was generated, and a content-free pause thought everywhere else.
luna chunk <|reserved_special_token_1|> luna reasoning <|reserved_special_token_2|>
filler chunk <|reserved_special_token_1|> 256x <|reserved_special_token_0|> <|reserved_special_token_2|>
The filler is 258 tokens. Chunk 0 is excluded… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled.Magpie-Reasoning-V2-250K-CoT-Llama3
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Reasoning-V2-250K-CoT-Llama3.llama-3.1-medprm-reward-training-set
Med-PRM-Reward (Version 1.0)
🚀 Med-PRM-Reward is among the first Process Reward Models (PRMs) specifically designed for the medical domain. Unlike conventional PRMs, it enhances its verification capabilities by integrating clinical knowledge through retrieval-augmented generation (RAG). Med-PRM-Reward demonstrates exceptional performance in scaling-test-time computation, particularly outperforming majority‐voting ensembles on complex medical reasoning tasks. Moreover, its… See the full description on the dataset page: https://huggingface.co/datasets/dmis-lab/llama-3.1-medprm-reward-training-set.llama3.2-3b-ultrafeedback-armorm-binarizedThis repository is associated with the paper Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model.
Code: https://github.com/DtYXs/Pre-DPO
llama3.2-java-codegen-90sft-10meta-claude-v1
LLaMA 3.2 Java Code Generation Dataset (90% SFT, 10% Meta Annotated with Claude)
This dataset contains 100,000 examples for Java method generation based on natural language instructions. It is built from the CodeXGLUE text-to-code dataset and designed to support both pure supervised fine-tuning (SFT) and reflection-based meta-learning approaches using Claude 4 Sonnet as the critique model.
🚀 Trained Models
Two models have been trained on this dataset:
SFT Model:… See the full description on the dataset page: https://huggingface.co/datasets/Naholav/llama3.2-java-codegen-90sft-10meta-claude-v1.fineweb10B-llama3
FineWeb Dataset tokenized with LLaMA-3.2-1B tokenizer
This dataset contains the FineWeb dataset preprocessed and tokenized using the LLaMA-3.2-1B tokenizer.
Each .bin file contains:
A header of 256 int32 values
Token data as uint32 values
File format version: 2
Dataset structure:
fineweb_train_*.bin: Training shards
fineweb_val_*.bin: Validation shard
Llama-3.1-8B-Instruct-Infinity-Instruct-0625
Llama-3.1-8B-Instruct-Infinity-Instruct-0625
Dataset Description
This dataset is part of the LK-Speculators collection for speculative decoding research. It contains 660K prompt-response pairs designed for training draft models that are used alongside Llama-3.1-8B-Instruct as the target model. The dataset was created by generating responses to the prompts from Infinity-Instruct-0625 with meta-llama/Llama-3.1-8B-Instruct at temperature=1.
For more details on the training… See the full description on the dataset page: https://huggingface.co/datasets/nebius/Llama-3.1-8B-Instruct-Infinity-Instruct-0625.openhermes-2.5-llama3Its a fork of teknium/OpenHermes-2.5, scripts used for generating this
from datasets import load_dataset
dataset = load_dataset("teknium/OpenHermes-2.5")
def _return_header(message)-> str:
role = message["from"]
header = ""
if role == "system":
header = "system"
elif role == "gpt":
header = "assistant"
elif role == "human":
header = "user"
return header
def encode_header(message):
text = ''
text = text + "<|start_header_id|>"… See the full description on the dataset page: https://huggingface.co/datasets/sumandas/openhermes-2.5-llama3.llama-3.1-medprm-reward-test-set🚀 Med-PRM-Reward is among the first Process Reward Models (PRMs) specifically designed for the medical domain. Unlike conventional PRMs, it enhances its verification capabilities by integrating clinical knowledge through retrieval-augmented generation (RAG). Med-PRM-Reward demonstrates exceptional performance in scaling-test-time computation, particularly outperforming majority‐voting ensembles on complex medical reasoning tasks. Moreover, its scalability is not limited to… See the full description on the dataset page: https://huggingface.co/datasets/dmis-lab/llama-3.1-medprm-reward-test-set.Llama-3.3-70B-Instruct-Infinity-Instruct-0625
Llama-3.3-70B-Instruct-Infinity-Instruct-0625
Dataset Description
This dataset is part of the LK-Speculators collection for speculative decoding research. It contains 660K prompt-response pairs designed for training draft models that are used alongside Llama-3.3-70B-Instruct as the target model. The dataset was created by generating responses to the prompts from Infinity-Instruct-0625 with meta-llama/Llama-3.3-70B-Instruct at temperature=1.
For more details on the… See the full description on the dataset page: https://huggingface.co/datasets/nebius/Llama-3.3-70B-Instruct-Infinity-Instruct-0625.Llama-3.3-70B-Inst-awq_SafeRLHF
Llama-3.3-70B-Inst-awq Responses for RefAlign Safety Alignment
This dataset contains responses generated for the paper Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data, which introduces the RefAlign alignment algorithm.
Code Repository: https://github.com/mzhaoshuai/RefAlign
This dataset specifically consists of responses generated by the casperhansen/llama-3.3-70b-instruct-awq model, given the prompts from the… See the full description on the dataset page: https://huggingface.co/datasets/mzhaoshuai/Llama-3.3-70B-Inst-awq_SafeRLHF.llama3-ultrafeedback-bertscore-bart-large-mnli
RefAlign: LLM Alignment Dataset
This dataset is used in the paper Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data.
Code: https://github.com/mzhaoshuai/RefAlign
This dataset is modified from https://huggingface.co/datasets/princeton-nlp/llama3-ultrafeedback. We use the BERTScore to choose the chosen and rejected responses.
Item with key ['Llama3.3-70B-Inst-Awq'] is the reference answers generated by… See the full description on the dataset page: https://huggingface.co/datasets/mzhaoshuai/llama3-ultrafeedback-bertscore-bart-large-mnli.Med_LLaMa3_fine-tuning_dataset
Med-LLaMA3 — Medical Instruction Fine-Tuning Dataset
A large, unified medical instruction-tuning corpus (~1.65 million examples) compiled, cleaned, and
standardized from a diverse set of public medical sources. It is the training corpus used to fine-tune
the Med-LLaMA3 family (LLaMA-3.2 1B/3B and LLaMA-3.1 8B) in the paper “Med-LLaMA3: Advancing Medical
Question-Answering Through Parameter-Efficient Fine-Tuning of Large Language Models” (Applied Sciences,
2026). All sources were… See the full description on the dataset page: https://huggingface.co/datasets/MohamedAhmedAE/Med_LLaMa3_fine-tuning_dataset.dynamic_sonnet_llama3
Dynamic Sonnet - Llama3
Curated dataset for benchmarking LLM serving systems
In real-world service scenarios, each request comes with varying input token lengths.
Some requests generate only a few tokens, while others produce a significant number.
Traditional fixed-length benchmarks fail to capture this variability, making it difficult to accurately assess real-world throughput performance.
This dynamic nature of input token lengths is crucial as it directly affects key features of… See the full description on the dataset page: https://huggingface.co/datasets/squeezebits/dynamic_sonnet_llama3.clean-gsm8k-aug-llama3-1.2b
Clean GSM8K-Aug Llama 3.2 1B Instruct
This dataset replaces the reasoning steps and answers in
cs-giung/clean-gsm8k-aug
with responses generated by
meta-llama/Llama-3.2-1B-Instruct.
The questions and split ordering match source revision
793d3a4bdf2d00068547466a44295f3208bad701. The model and tokenizer revision
is 9213176726f574b556790deb65791e0c5aa438b6.
Dataset structure
Each example has the following schema:
{
"question": str,
"steps": list[str]… See the full description on the dataset page: https://huggingface.co/datasets/cs-giung/clean-gsm8k-aug-llama3-1.2b.dolma-v1_7-30B-tokenized-llama3-nanosetTokenized (Llama 3) verison of NousResearch/dolma-v1_7-30B as a Nanotron dataset split into 10 GB chunks.
To recombine,
cat dolma-v1_7-30B-nanoset-l3_input_ids.npy.* > dolma-v1_7-30B-nanoset-l3_input_ids.npy
Can also be used directly with numpy, for example
import numpy as np
dataset_buffer_mmap = np.memmap("dolma-v1_7-30B-nanoset-l3_input_ids.npy", mode="r", order="C", dtype=np.int32)
dataset_buffer = memoryview(dataset_buffer_mmap)
dataset_number_of_tokens = int(len(dataset_buffer))
assist-llm-function-calling-llama3-chat
Function Calling dataset for Assist LLM for Home Assistant
This dataset is generated by using other conversation agent pipelines as teachers
from the deivce-actions-v2 dataset.
This dataset is used to support fine tuning of llama based models.
See Device Actions for a notebook for construction of this dataset and the device-actions dataset.
luna-reason-only.k-8.statml-arxiv-llama32
luna-reason-only.k-8.statml-arxiv-llama32
Prefix-only "thoughts" for next-token prediction on stat.ML arXiv LaTeX. Each thought is
visible reasoning about the next 8 Llama-3.2 tokens after a cut, written without ever
seeing that continuation. Intended to be spliced into the document before the chunk so a
small model (Llama 3.2 3B base) can read the reasoning and predict the chunk.
Complete: every designated chunk has a thought.
split
thoughts
coverage
documents… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.Llama-3.3-70B-Inst-awq_ultrafeedback_1in3
Generated Reference Answers for Language Model Alignment
This dataset contains responses generated for the research presented in the paper Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data.
The paper introduces RefAlign, a versatile REINFORCE-style alignment algorithm that utilizes language generation evaluation metrics, such as BERTScore, between sampled generations and reference answers as surrogate rewards. This approach… See the full description on the dataset page: https://huggingface.co/datasets/mzhaoshuai/Llama-3.3-70B-Inst-awq_ultrafeedback_1in3.llama-3.1-8b-funding-extraction-sft-ablations
LLaMA 3.1 8B Funding Extraction SFT Ablations
Ablation study results for LoRA SFT of Meta LLaMA 3.1 8B Instruct on structured funding metadata extraction from scholarly text.
The model extracts four fields: funder_name, award_ids, funding_scheme, and award_title.
Key findings
Factor
Best config
Avg F1
Overall best
synthetic, twostage (2+1 epochs), LoRA r=64, lr=3e-5
0.588
Data type
Synthetic >> non-synthetic (+0.126 avg F1)
—
LoRA rank
r=64 > r=32 > r=16
—… See the full description on the dataset page: https://huggingface.co/datasets/cometadata/llama-3.1-8b-funding-extraction-sft-ablations.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo
Tokenized, tag-wrapped form of JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.
Each thought is wrapped as <|reserved_special_token_1|> … thought … <|reserved_special_token_2|> … last prefix token and stored both as text (thought_text) and as Llama 3.2
token ids (input_ids).
The trailing token is the document token immediately before the cut (input_ids[chunk_start_index - 1]), copied from the document rather… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags
Tokenized, tag-wrapped form of JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.
Each thought is wrapped as
<|reserved_special_token_1|>
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|reserved_special_token_2|>
and stored both as text (thought_text) and as Llama 3.2
token ids (input_ids).
Longest thought: 514 tokens — a training run's max_thought_length must be at least
this.
Intended to be PREPENDED to the… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags-explained
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.
Each thought is wrapped as
<|reserved_special_token_1|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|reserved_special_token_2|>
and stored both as text (thought_text)… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags-explained.coco-deceptive-clip-llama3.1-8b
COCO-Deceptive-CLIP-LLaMA-3.1-8B Training Dataset
🏆 This work is accepted to ACL 2025 (Main Conference).
Figure: Attack success rate (ASR) and caption diversity of our model on the COCO dataset, illustrating its ability to generate deceptive captions that successfully fool CLIP.
Dataset Details
This dataset provides instruction–response pairs formatted as short two-turn conversations:
The user message contains:
A given image caption.
A set of task… See the full description on the dataset page: https://huggingface.co/datasets/ahnpersie/coco-deceptive-clip-llama3.1-8b.clean-gsm8k-aug-llama3-3.2b
Clean GSM8K-Aug Llama 3.2 3B Instruct
This dataset replaces the reasoning steps and answers in
cs-giung/clean-gsm8k-aug
with responses generated by
meta-llama/Llama-3.2-3B-Instruct.
The questions and split ordering match source revision
793d3a4bdf2d00068547466a44295f3208bad701. The model and tokenizer revision
is 0cb88a4f764b7a12671c53f0838cd831a0843b95.
Dataset structure
Each example has the following schema:
{
"question": str,
"steps": list[str]… See the full description on the dataset page: https://huggingface.co/datasets/cs-giung/clean-gsm8k-aug-llama3-3.2b.think-sft-mini-llama3-3.2b
think-sft-mini-llama3-3.2b
An exact-deduplicated mixture of five reasoning datasets, filtered to retain the
lowest 90.00% instance perplexities under
meta-llama/Llama-3.2-3B-Instruct. The schema is id, source, question, steps, and answer.
Metric
Value
Input rows after deduplication
854,432
Retained rows
768,988
Filtered low-likelihood rows
85,444
Per-token log-likelihood cutoff
-2.26903
Maximum retained perplexity
9.67006
Source composition… See the full description on the dataset page: https://huggingface.co/datasets/cs-giung/think-sft-mini-llama3-3.2b.Alpaca-Llama3.1-KD
Dataset Card for Alpaca-Llama3.1-KD
This dataset was introduced in the paper SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices.
The official code repository can be found here: ernlavr/SigmaScale.
Dataset Summary
This dataset is a distilled version of the classic tatsu-lab/alpaca dataset. It utilizes Meta-Llama-3.1-8B-Instruct as an answer generation model to generate high-quality, instruction-following responses for… See the full description on the dataset page: https://huggingface.co/datasets/ernlavr/Alpaca-Llama3.1-KD.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags
Tokenized, tag-wrapped form of JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.
Each thought is wrapped as <|reserved_special_token_1|> … thought … <|reserved_special_token_2|> and stored both as text (thought_text) and as Llama 3.2
token ids (input_ids).
Provenance
Source thoughts: JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32 — prefix-only reasoning about the next 8 Llama-3.2 tokens of
stat.ML… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags.