arasyi/quantum-api-drift

Quantum API Drift

Quantum API Drift is an evaluation benchmark for measuring whether LLM-generated quantum code targets the requested Qiskit SDK version. It accompanies the paper Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions.

The benchmark evaluates version fidelity, cross-version compatibility, failure modes, and documentation-guided repair across Qiskit 0.43, 1.3, and 2.0.

Dataset Configurations

benchmark

The default configuration contains 50 evaluation tasks. Each row has:

• —id: the upstream Qiskit HumanEval task identifier
• —description: the natural-language coding task
• —entry_point: the function the generated code must define
• —test_call: executable assertions used by the benchmark harness

Load it with:

from datasets import load_dataset

dataset = load_dataset("arasyi/quantum-api-drift")
tasks = dataset["test"]

version_anchored

This configuration contains 150 prompts: one prompt for each combination of 50 tasks and three requested SDK versions. Each row has:

• —id: the source task identifier
• —version: v0, v1, or v2
• —entry_point: the required function name
• —test_call: executable assertions
• —prompt: the complete SDK-version-anchored generation prompt

The version labels map to Qiskit releases as follows:

Load it with:

from datasets import load_dataset

dataset = load_dataset(
    "arasyi/quantum-api-drift",
    "version_anchored",
)
prompts = dataset["test"]

Auxiliary Files

The migration_notes/ directory contains the migration guidance used by the documentation-guided repair experiments:

• —v0_to_v1.txt: Qiskit 0.43 to 1.3
• —v1_to_v2.txt: Qiskit 1.3 to 2.0

These files are research artifacts and are not loaded as dataset rows.

Source and Modifications

The task descriptions and identifiers are derived from the Qiskit HumanEval dataset and its source repository, which are distributed under the Apache License 2.0.

Quantum API Drift modifies the upstream material by selecting 50 tasks, adapting executable test calls for cross-version evaluation, and constructing SDK-version-specific prompts. The modification and attribution notice is also provided in NOTICE.

Intended Use

This dataset is intended for evaluating version-aware quantum code generation, cross-version execution compatibility, API-drift failure modes, and documentation-guided repair. All rows are evaluation data and are published in the test split; they should not be presented as a training split.

Limitations

• —The benchmark measures API-level and execution-level validity, not full semantic circuit correctness.
• —Results depend on the execution environment and pinned Qiskit versions.
• —Repair results depend on the supplied migration notes.
• —The source benchmark and this derivative are public, so evaluations should discuss possible benchmark contamination.
• —Model generations, execution traces, and aggregate paper results are not included in this dataset repository.

License

The dataset is distributed under the Apache License 2.0. See LICENSE and NOTICE for the complete terms and attribution.

Citation

@misc{rasyidi2026benchmarkingapidrift,
  title={Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions},
  author={Rasyidi, Mohammad Arif and Faiz, Syahirul},
  year={2026},
  eprint={2607.04072},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  doi={10.48550/arXiv.2607.04072},
  url={https://arxiv.org/abs/2607.04072}
}

When reusing the underlying tasks, also cite the Qiskit HumanEval project.