chatdeepai/deepseek-1m-context-benchmark

DeepSeek 1M Context Benchmark

This dataset is the publication-safe measurement release for DeepSeek 1M Context Benchmark: Retrieval Accuracy, Latency, and Cost, version v1.0.0. It contains 344 sanitized terminal API records produced by the frozen protocol deepseek-v4-long-context-retrieval-v1.1.0 during a bounded run from 2026-08-06T20:17:02.706Z through 2026-08-07T00:07:44.737Z.

The study compared deepseek-v4-flash and deepseek-v4-pro on deterministic synthetic English retrieval and synthesis tasks. Provider-counted prompt tiers covered 32K, 128K, 512K, and approximately 950K tokens. The primary matrix varied model, prompt tier, objective task family, target position, and repeat while holding the request and grader contracts fixed.

• —Versioned source: GitHub release v1.0.0
• —Reproducible repository: Seek-Chat/deepseek-1m-context-benchmark
• —Research article: DeepSeek 1M Context Benchmark
• —Archival DOI: 10.5281/zenodo.21838863

What is included

The 344 terminal rows are divided into three analysis roles that must not be combined indiscriminately:

Only the 288 rows with analysis_role=primary_accuracy enter the published primary accuracy denominator. The pilot and India-vantage records are retained for auditability and separate bounded analyses.

The release contains sanitized measurements and hashes. It does not publish raw prompts, raw request bodies, raw responses, raw SSE payloads, credentials, authorization material, cloud resource identifiers, account data, or private URLs.

Configurations

Every configuration uses the Hugging Face test split because these are measured benchmark outcomes, not training examples.

The original data/all-attempts.jsonl file remains available as a 344-row raw alternate serialization of all_attempts_csv. It is intentionally excluded from the Viewer configuration list so every displayed configuration uses the repository's tabular CSV schema. Do not concatenate the CSV and JSONL copies. Empty fields mean that the provider or adapter did not supply a value; no missing values were silently imputed. See DATA-DICTIONARY.md for the 51-field case schema.

Loading the dataset

Load any published Viewer configuration with the final Hugging Face dataset ID:

from datasets import load_dataset

dataset_id = "chatdeepai/deepseek-1m-context-benchmark"

# The 288-case primary benchmark matrix.
primary = load_dataset(dataset_id, "primary", split="test")

# All 344 terminal rows in CSV form.
all_attempts = load_dataset(dataset_id, "all_attempts_csv", split="test")

# The bounded India client-network-vantage subset.
india = load_dataset(dataset_id, "india_latency", split="test")

For a local checkout, the underlying files can also be loaded directly with the csv or json builders.

Benchmark design

The frozen 288-case primary matrix is:

• —2 exact API model IDs: deepseek-v4-flash and deepseek-v4-pro;
• —4 provider-counted prompt tiers: 32K, 128K, 512K, and approximately 950K;
• —4 objective task families: single-record retrieval, two-record join, latest-version conflict resolution, and scattered event ordering;
• —3 target-position labels: beginning, middle, and end;
• —3 deterministic fixture repeats.

The benchmark used streaming Chat Completions, non-thinking mode, temperature 0, JSON Output, a 256-token study output cap, and a strict family-specific grader. Exact match required the expected key set, types, and values with no surrounding prose. The 256-token value is this study's generation cap, not a claim about the provider's maximum supported output.

The 20 pilot calls are excluded from all primary denominators. The 36 India-vantage calls form a separate matched client-network-vantage check at the 32K and approximately 950K edges; they do not represent Indian users or identify provider hosting locations.

Selected versioned results

These figures describe release v1.0.0 only:

• —Primary strict exact match: 152/288 (52.78%).
• —V4 Flash strict exact match: 71/144 (49.31%).
• —V4 Pro strict exact match: 81/144 (56.25%).
• —Primary valid JSON and exact key-set rates: 288/288 (100%) each.
• —Primary end-to-end stream time: 12,748 ms p50 and 162,295 ms p95 across the complete primary matrix.
• —Dated primary cache-miss cost upper bound: USD 33.589136, using the official price snapshot frozen on 2026-08-06 and complete returned usage for 288/288 rows.

These results are task-specific. They do not establish a universal model winner, and the dated cost figures are not current price quotes.

Provenance and integrity

The public release was built through an explicit allowlist from a validator export that reconciled all 344 planned terminal rows. The trusted publication-safe source archive has SHA-256:

38337e1213e2cf7ba7de8f21703e11772eddb71326361a66f27f7a8d0c114dcb

The following files preserve the versioned methodology and release evidence:

• —`provenance/protocol.json`: frozen experimental contract.
• —`provenance/calibration.json`: deterministic prompt-tier calibration.
• —`provenance/methodology.json`: publication methodology and analysis boundaries.
• —`provenance/summary.json`: canonical versioned aggregate metrics.
• —`provenance/prices.json`: dated price snapshot used for reported estimates.
• —`provenance/pilot-evidence-summary.json`: public-safe pilot and preflight disclosure.
• —`provenance/run-manifest.json` and `provenance/validator-export-manifest.json`: frozen run and export evidence.
• —`provenance/upstream-manifest.json`, `provenance/upstream-qa-report.json`, and `provenance/upstream-checksums.sha256`: integrity records for the complete GitHub release. The upstream checksum list also names charts and tables that are intentionally not duplicated in this compact Hugging Face bundle.
• —`CITATION.cff`: machine-readable citation metadata bound to the versioned Zenodo DOI.
• —`MANIFEST.json` and `QA.json`: integrity and safety checks for this Hugging Face upload bundle itself.

The full fixture generator, grader, publication code, charts, and release tables remain in the versioned GitHub repository.

Intended uses

This dataset is suitable for:

• —auditing the published primary accuracy calculation;
• —reanalyzing accuracy by model, prompt tier, task family, and target position;
• —inspecting transport, streaming, token, cache, and dated cost telemetry;
• —reproducing published tables or creating new visualizations;
• —studying matched outcomes and bounded client-network-vantage latency differences;
• —teaching reproducible benchmark reporting, denominator separation, and release provenance.

Out-of-scope uses

This dataset should not be treated as:

• —a training or fine-tuning corpus;
• —a general intelligence, coding, factuality, safety, multilingual, agent, or web-search benchmark;
• —a representative sample of real user prompts;
• —evidence of provider server location;
• —a recurring reliability or availability monitor;
• —a current pricing source;
• —a source of raw prompts or raw model outputs.

Limitations and potential sources of bias

• —All benchmark records are deterministic synthetic English data.
• —The strict JSON grader measures narrow exact retrieval and synthesis behavior, not open-ended response quality.
• —The run covers one bounded execution window and two exact model IDs as routed during that window.
• —Target-position semantics differ by task family, so aggregate position comparisons require the detailed methodology.
• —Latency includes the disclosed client network vantage and workload; it should not be generalized to every user, region, or date.
• —The India subset is intentionally small and limited to matched single-record cases at two prompt tiers.
• —Provider pricing, model aliases, routing, and fingerprints can change after this release.
• —Hashes of withheld raw artifacts support provenance but do not make the raw content public.

Licensing and attribution

The dataset is licensed under Creative Commons Attribution 4.0 International. See LICENSE-DATA.txt. Attribute Seek-Chat, identify version v1.0.0, link the versioned release, and retain material methodology and limitation notices when redistributing or adapting the data.

Provider names and trademarks belong to their respective owners. This is an independent benchmark and is not an official DeepSeek dataset.

Citation

Use the versioned archival DOI 10.5281/zenodo.21838863:

@dataset{seek_chat_2026_deepseek_1m_context,
  author    = {Seek-Chat},
  title     = {DeepSeek 1M Context Benchmark: Retrieval Accuracy, Latency, and Cost},
  year      = {2026},
  version   = {1.0.0},
  doi       = {10.5281/zenodo.21838863},
  url       = {https://github.com/Seek-Chat/deepseek-1m-context-benchmark/releases/tag/v1.0.0}
}

The archival record resolves at https://doi.org/10.5281/zenodo.21838863.