Datasetpublic

Santhiyarajan/omission-detection-synthetic

Omission Detection — Synthetic Sweep What is Omission Detection? Large language models (LLMs) in agentic pipelines often omit information present in their context window — they fail to surface a relevant fact even when it is theoretically visible. This dataset captures 75,876 controlled trials designed to measure and attribute these omissions across 9 taxonomic layers (L0–L8). Each trial generates a synthetic clinical document, embeds a "needle" fact at a… See the full description on the dataset page: https://huggingface.co/datasets/Santhiyarajan/omission-detection-synthetic.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes10downloads

Dataset Card

Omission Detection — Synthetic Sweep

What is Omission Detection?

Large language models (LLMs) in agentic pipelines often omit information present in their context window — they fail to surface a relevant fact even when it is theoretically visible. This dataset captures 75,876 controlled trials designed to measure and attribute these omissions across 9 taxonomic layers (L0–L8).

Each trial generates a synthetic clinical document, embeds a "needle" fact at a controlled depth, passes the document through deterministic pipeline layers (redaction, pagination, chat templating, orchestration), and optionally runs model inference to see whether the needle is retrieved.

Experiment Design

component	description
Documents	Synthetic clinical notes generated from 8 noise-sentence pools (symptom reports, nursing notes, drug administration, imaging findings, etc.)
Needles	Injected facts using 3 families: literal, paraphrase, conflict (8 templates each). Each needle has a unique canary string.
Models	5 architectures: Full Attention, SWA-Hybrid, SSM-Hybrid, Gemma MoE, Qwen Linear MoE
Engines	llama.cpp (CPU quantised) and vLLM (GPU)
Layers (omission taxonomy)	L0: redaction, L1: pagination, L2: chat template, L3: orchestration prompt, L4: sampler, L5: attention window, L6: position encoding, L7: MoE routing, L8: tool loop
Phases	A (pipeline audit), B/C (model inference), D (distractor sweep), E (orchestrator variation), F (framework: LangChain/ADK)

Configs

Each JSONL file is one (model, engine) config — load with:

python

from datasets import load_dataset

ds = load_dataset(
    "santhiyarajan/omission-detection-synthetic",
    "full_attn__llama_cpp",
    split="train",
)
print(len(ds))  # 13437

Config	Model	Engine	Rows
fullattnllamacpp	Full Attention	llama.cpp	13,437
fullattn_vllm	Full Attention	vLLM	8,309
gemmamoellamacpp	Gemma MoE	llama.cpp	10,223
gemmamoe_vllm	Gemma MoE	vLLM	5,342
qwenlinearmoe_llamacpp	Qwen Linear MoE	llama.cpp	6,955
qwenlinearmoe__vllm	Qwen Linear MoE	vLLM	3,941
ssmhybridllamacpp	SSM-Hybrid	llama.cpp	9,598
swahybridllamacpp	SWA-Hybrid	llama.cpp	10,235
swahybrid_vllm	SWA-Hybrid	vLLM	7,436
phaseffull_attn	Full Attention	llama.cpp (F)	50
phasefgemma_moe	Gemma MoE	llama.cpp (F)	50
phasefssm_hybrid	SSM-Hybrid	llama.cpp (F)	50
phasefswa_hybrid	SWA-Hybrid	llama.cpp (F)	50
phasefcomparison	All (cross-model)	llama.cpp (F)	200

Fields

column	type	description
trial_id	string	unique identifier (phase + index)
phase	string	A (pipeline audit), B/C (model inference), D (distractor), E (orchestrator), F (framework)
family	string	needle family: literal, paraphrase, conflict
needle_id	string	needle identifier
canary	string	unique canary string embedded in the needle
depth	float	needle injection depth (0.0–1.0, fraction of document)
distractors	bool	whether distractor values were added
size_tokens	int	target document size in tokens
page_tokens	int	tokens per page (for pagination layer)
redaction	string	redaction mode: off, deid, aggressive
fetch_policy	string	pagination page-fetch policy
truncate_limit	int	character truncation limit (0 = no truncation)
escape_special	bool	whether special tokens (<startofturn> etc.) were escaped
engine_events	string	JSON list of engine events (token counts, timestamps)
response	string	model's answer text (empty string for Phase A — no inference)
retrieved	bool	whether response matched the expected answer
latency_s	float	model inference latency (-1.0 if no inference occurred)
attribution	string	omission classification: success, unsurfaced*, software, model_
logit_probe	string	JSON logit probe values (empty string if not probed)
seed	int	reproducibility seed
doc_text	string	full synthetic clinical document text (the model's context window)
expected_answer	string	the correct answer embedded in the needle
iterations	int	number of inference iterations
framework	string	none (sweep), langchain, adk (Phase F)
source	string	synthetic
domain	string	clinical
engine	string	llama_cpp, vllm
tool_provider	string	mockmcp, docmcp
max_iterations	int	maximum tool-call iterations

Null-value convention

Because Phase A trials have no model inference, some fields use sentinel values:

—response: "" (empty string) when no model was called
—retrieved: False when not applicable
—latency_s: -1.0 when no inference occurred
—logit_probe: "" (empty string) when not probed

Citation

bibtex

@inproceedings{rajan2026omission,
  title={Where Does the Information Go? Attribution of Omissions in
         Agentic LLM Pipelines},
  author={Rajan, Santhiya and ...},
  booktitle={Under review},
  year={2027}
}

License

CC-BY-4.0