shanaka95/enterprise-rag-questions

shanaka95/enterprise-rag-questions

A focused training corpus for domain-adapted embedding models targeting retrieval over enterprise documents. Each row is a (doc_id, question) pair where the question is synthetically generated by an instruction-tuned LLM (Gemma-4-12B-it-qat) from a corresponding source document.

The dataset is derived from the public benchmark `onyx-dot-app/EnterpriseRAG-Bench`. For each document in the source corpus we generate five diverse, answerable training questions. Together they cover 511,955 enterprise documents and 2,560,220 questions across the full documents split of the source benchmark.

Why this dataset exists

Public retrieval benchmarks typically ship with a small evaluation set (e.g. ~1k questions) which is great for measuring retrieval quality but insufficient for fine-tuning an embedding model: contrastive and distillation training need hundreds of thousands to millions of (query, document) pairs in the target domain.

enterprise-rag-questions fills that gap by providing 5×10⁶ query-document pairs that share the exact same document distribution, vocabulary, and writing style as onyx-dot-app/EnterpriseRAG-Bench/test. A retrieval model fine-tuned on this corpus will see in-distribution text during training and can be evaluated zero-shot on the source benchmark's questions test split.

Use cases

1. 1.Fine-tuning embedding models for enterprise retrieval — pair every (doc_id, question) row with the corresponding document text from the source dataset to form (query, positive) pairs. Sample in-batch or mined negatives from the same corpus.
2. 2.Hard-negative mining — the dense, domain-matched document pool makes this corpus ideal as the candidate pool for hard-negative mining (BM25 negatives, dense ANN negatives).
3. 3.Knowledge-distillation of retrievers — use the questions as teacher query targets for distillation pipelines.
4. 4.Synthetic-query augmentation for any retrieval pipeline over enterprise text — the questions are stylistically diverse (factual, comparative, conditional, procedural, quantitative) and reflect the kinds of questions a real analyst would ask of long technical documents.

Schema

A row is one (doc_id, question) pair. There are exactly 5 questions per doc_id for 511,860 documents (the remaining 95 documents have 1, 3, 7, 8, or 10 questions due to end-of-stream partial generation; filter as needed via df.groupby('doc_id').filter(lambda g: len(g) == 5)).

How it was built

1. 1.Stream the documents split of onyx-dot-app/EnterpriseRAG-Bench (511,957 docs).
2. 2.For each document, prompt a local vLLM-served google/gemma-4-12B-it-qat-w4a16-ct model with a strict-JSON schema that requires exactly five questions.
3. 3.Validate every response against the JSON schema and discard batches whose questions fail parsing. The validated questions are stored in SQLite with a (doc_id, question) unique index to guarantee one canonical row per pair.
4. 4.Concatenate two snapshots of the pipeline run (early run covering ~196k documents and late run covering the remaining ~316k) and deduplicate on (doc_id, question) to produce the final training corpus.

Generation prompt and few-shot examples are reproducible from the source dataset and release-ready model.

How to use

from datasets import load_dataset

ds = load_dataset("shanaka95/enterprise-rag-questions", split="train")
print(ds[0])
# {'doc_id': 'dsid_...', 'question': 'Under what specific conditions is a model artifact ...'}

# Join with the source documents to get the positive passage:
docs = load_dataset("onyx-dot-app/EnterpriseRAG-Bench", "documents", split="test")
doc_text = {d["doc_id"]: d["content"] for d in docs}
ds = ds.map(lambda r: {"passage": doc_text[r["doc_id"]]})

Source / citation

This dataset is a derived work. Cite the original benchmark:

@dataset{enterprise_rag_bench,
  title  = {EnterpriseRAG-Bench},
  author = {Onyx},
  year   = {2026},
  url    = {https://huggingface.co/datasets/onyx-dot-app/EnterpriseRAG-Bench},
  note   = {MIT License, arXiv:2605.05253}
}

The questions in this dataset are synthetically generated by Gemma-4-12B-it-qat-w4a16-ct and are released under the same MIT license as the source.

License

MIT (inherited from onyx-dot-app/EnterpriseRAG-Bench).