qcri-ai/HCTQA
HCT-QA: Human-Centric Tables Question Answering HCT-QA is a benchmark dataset designed to evaluate large language models (LLMs) on question answering over complex, human-centric tables (HCTs). These tables often appear in documents such as research papers, reports, and webpages and present significant challenges for traditional table QA due to their non-standard layouts and compositional structure. The dataset includes: 2,188 real-world tables with 9,835 human-annotated QA… See the full description on the dataset page: https://huggingface.co/datasets/qcri-ai/HCTQA.
HCT-QA: Human-Centric Tables Question Answering
HCT-QA is a benchmark dataset designed to evaluate large language models (LLMs) on question answering over complex, human-centric tables (HCTs). These tables often appear in documents such as research papers, reports, and webpages and present significant challenges for traditional table QA due to their non-standard layouts and compositional structure.
The dataset includes:
- 2,188 real-world tables with 9,835 human-annotated QA pairs
- 4,679 synthetic tables with 67,500 programmatically generated QA pairs
- Logical and structural metadata for each table and question
📄 Paper: [Title TBD] The associated paper is currently under review and will be linked here once published.
How to load in Python (as pandas DataFrames):
from datasets import load_dataset
import pandas as pd
dataset = load_dataset("qcri-ai/HCTQA")
# Convert each split to a pandas DataFrame
train_df = pd.DataFrame(dataset['train'])
val_df = pd.DataFrame(dataset['validation'])
test_df = pd.DataFrame(dataset['test'])📊 Dataset Splits
🏆 Leaderboard
📌 If you're evaluating on this dataset, open a pull request to update the leaderboard.
Dataset Structure
Each entry in the dataset is a dictionary with the following structure:
Sample Entry
{
"table_id": "arxiv--1--1118",
"dataset_type": "arxiv",
"table_data": {
"table_as_csv": ",0,1,2\n0,Domain,Average Text Length,Aspects Identified\n1,Journalism,50,44\n...",
"table_as_html": "<table><tr><th>Domain</th><th>Average Text Length</th>...",
"table_as_markdown": "| Domain | Average Text Length | Aspects Identified |...",
"table_image_local_path_within_github_repo": "tables/images/arxiv--1--1118.jpg",
"table_image_url": "https://hcsdtables.qcri.org/datasets/all_images/arxiv_1_1118.jpg",
"table_properties_metadata": {
"Standard Relational Table": true,
"Row Nesting": false,
"Column Aggregation": false
}
},
"questions": [
{
"question_id": "arxiv--1--1118--M0",
"question": "Report the Domain and the Average Text Length where the Aspects Identified equals 72",
"question_template_for_synthetic_only": "Report [column_1] and [column_2] where [column_3] equals [value]",
"question_properties_metadata": {
"Row Filter": true,
"Aggregation": false,
"Returned Columns": true
},
"answer": "{Psychology | 86} || {Linguistics | 90}",
"prompt": "<system>...</system><user>...</user>",
"prompt_without_system": "<user>...</user>"
}
]
}Ground Truth Format
<b> Explain the GT format here </b> Example: {value1 | value2} || {value3 | value4}
Table Properties
For details on table and question properties please see our paper