ollamaweights/synthetic-dataset-1208

Synthetic Key-Value Retrieval 32K

This is a deterministic synthetic benchmark for exact key-value retrieval from a long context. It is designed for evaluating long-context inference and KV cache compression methods.

Context format

The context contains a one-time task description followed by an array:

You are given an array of key-value entries. Every key begins with K_ and every value begins with V_. Each entry has the format [key: value]. Given a query key, find its exact match and return only the corresponding value, exactly as written.
[
[K_1C80A3B1799D: V_BDD606671AD1],
[K_3EB146685257: V_23B8392456DE],
[K_1A3DBC8960A9: V_BD9CAD3C2D6D],
...
]

Each question has the form:

Query key: K_1C80A3B1799D

The expected answer is:

V_BDD606671AD1

The evaluator supplies Answer: immediately before generation. This is a generation cue; the reference answer remains only the expected V_... value.

Compact storage and evaluation samples

The test split stores the long context once. The questions and answers arrays are positionally aligned: questions[i] is answered by answers[i]. An evaluator should expand the compact row into one sample per aligned question/answer element while reusing the same context.

• —Stored contexts: 1
• —Key-value entries: 1,182
• —Evaluation samples after expansion: 1,182
• —Raw context tokens: 31,945 with Qwen/Qwen3-8B, measured using add_special_tokens=False
• —Seed: 42
• —Key and value payloads: unique 12-character uppercase hexadecimal strings
• —Maximum generation tokens: 32

Columns

For compression comparisons, first measure the uncompressed baseline and keep the context format, tokenizer, chat template, and query-aware setting fixed across methods.