anonymous-submission042/RFSchemBench

RFSchemBench

A multimodal LLM evaluation benchmark for radio-frequency circuit schematic understanding, organized by a four-level semantic hierarchy:

1. 1.Component Understanding — visible component, parameter, label, and supply-rail recognition.
2. 2.Structural Understanding — net membership, pin-to-net mapping, boundary connectivity, and pair-via-net topological reasoning.
3. 3.Functional Understanding — circuit functional role, signal-form classification, supply strategy, sub-type identification.
4. 4.Dynamic Reasoning — counterfactual plot choice and schematic-modification ↔ simulation-result matching, grounded in ngspice simulation.

The benchmark contains 2,348 questions across 590 rendered schematic pages from publicly available RF schematic data.

Quick start

from datasets import load_dataset

# Permissive subset (CC-BY-4.0; recommended for most users)
ds = load_dataset("anonymous-submission042/RFSchemBench", "permissive", split="test")

# Full benchmark including a NonCommercial-ShareAlike subset
ds_nc = load_dataset("anonymous-submission042/RFSchemBench", "nc_allowed", split="test")

print(ds[0]["question"], "→ answer:", ds[0]["answer"])
ds[0]["image"].show()  # PIL.Image of the schematic

Configurations

Schema

Each row has the following fields:

Construction

The benchmark is constructed via expert-rule-guided programmatic generation from authoritative sources:

• —Domain experts encode question-generation rules and gold-answer semantics into Python programs.
• —Gold answers are extracted deterministically from authoritative source artifacts (KiCad CLI output, Qucs native schematic graph, ngspice simulation outputs).
• —LLMs are deliberately excluded from the gold-answer path; they are used only as an auxiliary RF-relevance gate at the page level.
• —An iterative rule-refinement loop catches edge cases during construction; the released gold answers reflect the latest revisions.

This avoids the gold-answer noise floor of LLM-as-Generator benchmarks while scaling beyond purely human-curated efforts.

License

This dataset is released under a two-tier license model because the upstream sources have heterogeneous licenses:

• —`permissive` config (recommended default): all rows under CC-BY-4.0. Compatible with commercial use, redistribution, and derivative works subject to attribution.
• —`nc_allowed` config: includes one source class (m17 digital-radio community hardware, 90 questions) which is upstream-licensed under CC-BY-NC-SA-4.0 (NonCommercial-ShareAlike). Per-row license field marks affected items. Users must respect NC + ShareAlike for those rows.

Per-source licensing summary:

For redistribution that requires fully permissive licensing, use only the permissive config.

Limitations

1. 1.Source-class size imbalance: question counts per source class span 40–974; per-source claims should be reported with N.
2. 2.Dynamic Reasoning scope: only one source class has the simulation-grounded subset (55 questions). This dimension is reported as a small stress test, not the main result.
3. 3.Language: questions are evaluated in English. (A Chinese parallel set was used internally during construction for human review but is not part of the released schema.)
4. 4.Single-image protocol: each question is paired with one primary schematic image (Dynamic Reasoning rows additionally provide context plots / option plots).
5. 5.Anonymized release: this submission account is for double-blind peer review. The dataset will be transferred to the official maintainer account upon acceptance.

Citation

@misc{rfschembench2026,
  title  = {RFSchemBench: A Multi-Source, Hierarchically-Structured Multimodal Benchmark for RF Circuit Schematic Understanding},
  author = {Anonymous},
  year   = {2026},
  note   = {Submitted to NeurIPS 2026 Evaluations \& Datasets Track}
}

Contact

For benchmark integrity issues (gold-answer corrections, RF-gate disputes, parser / scorer concerns), please open a Discussion on this dataset's HuggingFace page. During the double-blind review window, identifying contact details are intentionally withheld.