helkhour/web-search-router-labels

Web-Search Need Labels (draft, validation-scale)

Status: draft. This is the audited validation-scale release (517 questions) of a dataset that labels open-domain questions by whether a mid-size LLM needs web search to answer them correctly. A train-split scale-up (~509k questions) is in progress and will be released separately.

Built as part of a semester project on a web-search MCP service and shared Elasticsearch indexing infrastructure for the Swiss AI initiative (swiss-ai/search-index).

What the label means

For each question, the target model (Qwen3-32B) sampled k=5 answers in two modes: from parametric memory only (no_search) and with web-search evidence in the prompt (with_search, Exa backend). Every answer was judged against the reference answers by an LLM judge (Apertus-70B), cross-checked by an independent discriminative reranker (Qwen3-Reranker-8B, ref_can protocol), with disagreements adjudicated by a stronger LLM referee.

• —web_search_needed = true — the model failed all k no-search samples but passed with search evidence (pass@k rule). 87 questions.
• —web_search_needed = false — the model already answers from memory (or search did not help). 429 questions.
• —web_search_needed = null (label_status = "unresolved") — all relevant samples errored; quarantined rather than labeled. 1 question.

Splits

Row schema (combined)

• —id, question, references (gold answers), source (origin dataset/split)
• —filter — quality-filter metadata (Llama-3.3-70B): candidate_type (static / search_likely_static), keep, reason
• —no_search / with_search — the k sampled answers with per-sample verdicts, and the mode-level pass_at_k
• —web_search_needed, label_status

No scraped web page content is included; with_search.search_results is empty by construction in this export.

Curation and audit (summary)

Raw pool 715 questions → 549 after an LLM quality filter (drops subjective, contradictory, time-drifting, yes/no rows) → 543 sampled and run → 517 curated after two human-audit loops removed 26 questions with corrupted references. Judge verdicts were audited end-to-end: empty-answer false-accepts quarantined, a lexical guard used as an audit instrument, 85% per-sample agreement between judge and reranker, and referee adjudication of all disputed answers. Clean pass@k: 0.76 (no search) → 0.89 (with search); essentially all of the lift comes from NQ-Open. The needed/not-needed labels where both verifiers agree (55 needed / 403 not-needed) form the high-confidence core.

Known limitations: the pool is easy and NQ-heavy (97% static under the LLM filter), so the ~17% needs-search rate does not generalize to arbitrary web questions; verdicts come from mid-size open models, audited but not human-labeled at scale.

Sources and licensing

Questions and references derive from:

• —Natural Questions Open (google-research-datasets/nq_open) — CC BY-SA 3.0.
• —MS MARCO (QnA) — free for non-commercial research purposes per the MS MARCO terms; Microsoft's terms govern the underlying questions/answers.

Model outputs were generated with Qwen3-32B; verdicts with Apertus-70B and Qwen3-Reranker-8B. Rows carry a source field so subsets can be filtered by origin and license. Use of this dataset must respect the source datasets' terms; the MS MARCO-derived rows are for non-commercial research use.

Intended use

Training and evaluating when-to-search decision policies (e.g. RL decision targets CALL_WEB_SEARCH vs ANSWER_DIRECTLY for tool-augmented models), and benchmarking answer verifiers against the included per-sample verdicts.