gyung/ruler-niah-multilength-eval-benchmark

📌 Fixed Multi-Length RULER NIAH Benchmark (1K, 2K, 4K, 8K)

Deterministic synthetic Needle-In-A-Haystack (NIAH) benchmark splits for reproducible long-context evaluation.

Dataset Specifications:

• —Tasks (4):
• —niah_single_1: Repeat haystack, single word needle, number value.
• —niah_single_2: Essay haystack, single word needle, number value.
• —niah_single_3: Essay haystack, single word needle, UUID value.
• —niah_multikey_1: Essay haystack, 4 keys needle, number value.
• —Context Lengths (4): 1024 (1K), 2048 (2K), 4096 (4K), 8192 (8K)
• —Samples per Task per Length: 50 (Total 800 samples)
• —Tokenizer: TinyLlama/TinyLlama_v1.1
• —Random Seed: 42 (Guaranteed 100% bit-for-bit identical evaluation prompts)

Files in this dataset:

Each combination is saved as an individual .jsonl file: {task}_{length}.jsonl (e.g. niah_single_1_1024.jsonl, niah_multikey_1_8192.jsonl).

Schema:

{
  "task": "niah_single_1",
  "length": 1024,
  "length_label": "1K",
  "index": 0,
  "input": "... prompt text containing haystack and needle ...",
  "outputs": ["8090293"],
  "gen_prefix": ""
}