Datasetpublic

jet-ai/ruler-100-nemotron

RULER-100 — Nemotron-Nano-v3 tokenized RULER long-context evaluation data, regenerated with the nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 (instruct) tokenizer so the labeled context lengths are exact for that model — instead of drifting, as they do when RULER data tokenized for a different model (e.g. Qwen3) is fed to Nemotron. What's here 7 context lengths: 4096, 8192, 16384, 32768, 65536, 131072, 262144 (the model's max). 13 RULER tasks: niah_single_1/2/3… See the full description on the dataset page: https://huggingface.co/datasets/jet-ai/ruler-100-nemotron.

sourceHugging Facecc-by-sa-4.0updated 2mo agoView on Hugging Face

0likes73downloads

Dataset Card

RULER-100 — Nemotron-Nano-v3 tokenized

RULER long-context evaluation data, regenerated with the `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` (instruct) tokenizer so the labeled context lengths are exact for that model — instead of drifting, as they do when RULER data tokenized for a different model (e.g. Qwen3) is fed to Nemotron.

What's here

—7 context lengths: 4096, 8192, 16384, 32768, 65536, 131072, 262144 (the model's max).
—13 RULER tasks: niah_single_1/2/3, niah_multikey_1/2/3, niah_multivalue, niah_multiquery, vt, cwe, fwe, qa_1 (SQuAD), qa_2 (HotpotQA).
—100 samples per (length, task) → 91 files, 9 100 examples total.

Layout

<length>/data/<task>/validation.jsonl

Each line has four fields:

field	type	description
`index`	int	sample id within the (length, task) file
`input`	str	the full RULER task prompt including the answer prefix (no chat template)
`outputs`	list[str]	gold answer(s); RULER scores with substring match (`all` for niah/vt/cwe/fwe, `part` for qa)
`length`	int	tokenized length of the raw input under the Nemotron tokenizer

How it was generated (chat-template handling)

The prompts are stored raw (no chat template). Generation reserved 16 tokens of each length budget for the Nemotron-Nano-v3 chat wrapper that is applied at inference time, i.e. data was produced at max_seq_length = length − 16. This mirrors NVIDIA NeMo-Skills' nemo_skills/dataset/ruler/prepare.py (raw data, max_seq_length -= template_tokens).

At inference, wrap each input as a single user turn with thinking disabled:

python

tok.apply_chat_template(
    [{"role": "user", "content": row["input"]}],
    add_generation_prompt=True,
    enable_thinking=False,   # -> closed <think></think> so the model answers directly
)
# renders: <|im_start|>system\n<|im_end|>\n<|im_start|>user\n{input}<|im_end|>\n<|im_start|>assistant\n<think></think>

enable_thinking=False is required: the template defaults to True, which would spend RULER's short generation budget (32–128 tokens) on reasoning. With the wrapper applied, prompts land at ≤ length tokens for every sample (verified at 4096 and 262144).

Per-task generation budgets (tokens_to_generate): niah 128, vt 30, cwe 120, fwe 50, qa 32.

NIAH "essay" haystack — public-domain (Project Gutenberg)

The NIAH "essay" haystack uses a public-domain Project Gutenberg corpus (7 pre-1929 English novels; PG header/footer/trademark stripped) in place of the copyrighted Paul Graham essays. The 5 essay tasks (niah_single_2, niah_single_3, niah_multikey_1, niah_multivalue, niah_multiquery) were regenerated through the same pipeline as the rest of the dataset — validated to reproduce a non-essay task 100/100: RULER prepare.py (--model_template_type base, max_seq_length = bucket − 16, 100 samples, Nemotron tokenizer, seed 42), with the Gutenberg corpus fed to niah.py via its RULER_ESSAY_JSON override, then converted to this 4-field schema. Reproduction scripts are bundled in `scripts/`: `build_gutenberg.py`, `niah.py`, and `regen_nemotron_essays.py`. The other NIAH tasks use noise/needle haystacks (no third-party text).

Validation. All 3,500 regenerated essay records were checked exhaustively: 4-field schema, sequential index (0–99), length == Nemotron_tokenize(input) + 128 ≤ bucket − 16, correct answer count per task, every gold answer actually inserted as a needle (is: <value>), the answer-prefix tail intact, no Paul-Graham / "Project Gutenberg" trademark text, and each haystack confirmed drawn from the Gutenberg corpus. The same length == tokens + tokens_to_generate convention was cross-checked against the untouched non-essay files, confirming the regenerated tasks are consistent with the rest of the dataset.

Usage

python

from huggingface_hub import snapshot_download
local = snapshot_download("jet-ai/ruler-100-nemotron", repo_type="dataset")
# point a RULER harness data_dir at `local`; files are <length>/data/<task>/validation.jsonl

Licensing

Released under CC BY-SA 4.0 — the strongest obligation among the dataset's components (from the embedded SQuAD / HotpotQA passages). Every component is openly licensed, so the dataset is freely redistributable with attribution + ShareAlike. Per-source breakdown:

Component	Files	Source / holder	License
RULER pipeline & synthetic structure (task construction, prompts, NIAH/VT/CWE/FWE content)	all files	NVIDIA RULER	Apache-2.0
SQuAD v2 passages & questions	`*/data/qa_1/validation.jsonl`	SQuAD v2 (Rajpurkar et al., Stanford)	CC BY-SA 4.0 (attribution + ShareAlike)
HotpotQA passages & questions	`*/data/qa_2/validation.jsonl`	HotpotQA (dev distractor)	CC BY-SA 4.0 (attribution + ShareAlike)
Project Gutenberg novels (NIAH "essay" haystack)	`*/data/{niah_single_2, niah_single_3, niah_multikey_1, niah_multivalue, niah_multiquery}/validation.jsonl`	Project Gutenberg — pre-1929 English novels, PG header/footer/trademark stripped	Public domain

Notes:

—Attribution + ShareAlike: the QA files embed CC BY-SA 4.0 text, so attribute SQuAD / HotpotQA and keep adaptations under BY-SA. The RULER pipeline is Apache-2.0 and the Gutenberg haystack is public domain (PG's trademarked boilerplate is stripped, leaving only the public-domain works) — both are compatible with redistributing the collection under CC BY-SA 4.0.
—No rights-restricted content remains. The five essay-haystack NIAH tasks were the only ones that ever embedded third-party prose, and those now use the public-domain Project Gutenberg corpus (previously the all-rights-reserved Paul Graham essays). Every other task was left unchanged and never contained third-party text — its haystack is RULER-generated: niah_single_1 uses a repeated noise sentence, niah_multikey_2 / niah_multikey_3 use synthetic needle lines, and vt / cwe / fwe use synthetic chains / word lists.
—This breakdown is informational, not legal advice — confirm with your legal / IP office before release.

Provenance

Synthetic data built by NVIDIA RULER. Generation seed 42, 100 samples per (length, task), nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 tokenizer; NIAH "essay" haystack from a public-domain Project Gutenberg corpus. Reproduction scripts bundled in `scripts/`: build_gutenberg.py, niah.py (RULER, with the RULER_ESSAY_JSON override), and regen_nemotron_essays.py (the exact generate + 4-field-convert pipeline).

Added buckets: 98304, 163840, 196608 (2026-07-18/20)

Three buckets were generated AFTER the original publish, to fill the 64k→262k range for a RULER context-length sweep: `98304/`, `163840/`, `196608/`. Pipeline for all three: NVIDIA/RULER @ 38da79d7, the same patched niah.py + Gutenberg haystack + prepare.py --model_template_type base --num_samples 100 --max_seq_length <bucket−16> (98288 / 163824 / 196592), seed 42, Nemotron tokenizer — IDENTICAL to the original buckets. Fidelity gate: regenerating the published 65536 bucket with this exact pipeline reproduced all 13 task files byte-identical, so the new buckets sit on the same curve (each = 13 tasks × 100 samples, length ≤ bucket). Published to this HF dataset repo alongside the original lengths; the front-matter data_files glob (*/data/*/validation.jsonl) includes them automatically in the default config.