TPelc/Current_Trivia_Knowledge-benchmark

Current Trivia Knowledge RAG Benchmark

Short Summary:

A 140-QA pair (70 train, 70 test) dataset for real-world RAG evaluation. It features current knowledge questions unavailable to LLMs trained before 2024 (e.g., GPT-4o) across diverse domains, and includes human feedback for the training set, enabling robust assessment of contextual information's critical impact on LLM accuracy.

Introduction & Motivation:

This dataset addresses the critical need for a dynamic RAG benchmark. It provides a unique and controlled testing ground for RAG pipelines.

Data Collection and Generation:

GPT-4o generated QA pairs using web-sourced current information (2024-2025) to eliminate pre-existing LLM knowledge bias. The 70-pair training set includes human feedback on initial responses, while the 70-pair test set features novel QA pairs derived from feedback-enhanced training examples for robust generalization assessment.

Context Topics

Here are all topics used as a RAG context:

• —2024 Nobel Prizes
• —2024 Summer Olympics
• —2024 Eurovision
• —2025 MWC
• —2025 Academy Awards
• —US Presidents
• —Intel Key Figures

Dataset Structure & Features:

• train split (70 examples): question, answer, feedback (human-annotated).

• test split (70 examples): question, answer.

Intended Use Cases:

Ideal for:

• Real-world RAG Simulation: Benchmarking against dynamic, current knowledge.

• Feedback-Driven RAG Development: Training and evaluating RAG models using human feedback.

• Generalization & Knowledge Freshness: Assessing system adaptability to new questions and rapidly changing information.

Limitations and Biases:

Synthetic (GPT-4o generated) QA, Wikipedia-only contexts, specific feedback format, and limited size (140 pairs) are key considerations.