SahmBenchmark/financial-reports-extractive-summarization_eval

Financial Reports Extractive Summarization Evaluation Dataset

Validation and test splits for evaluating models on Arabic financial reports extractive summarization.

Dataset Structure

• —Format: Simple prompt-answer pairs
• —Validation: ~20 examples (10%)
• —Test: ~20 examples (10%)
• —Language: Arabic
• —Domain: Financial reports and market news

Fields

• —id: Unique identifier
• —prompt: The summarization prompt
• —full_text: Complete financial report
• —answer: Ground truth extractive summary
• —report_type: Type of report
• —file_name: Original file
• —split: 'validation' or 'test'
• —text_length: Full text length
• —summary_length: Summary length
• —compression_ratio: Compression percentage

Usage

from datasets import load_dataset

dataset = load_dataset("SahmBenchmark/financial-reports-extractive-summarization_eval")

# Access splits
val_data = dataset['validation']
test_data = dataset['test']

# For evaluation
for example in test_data:
    model_output = model.generate(example['prompt'])
    ground_truth = example['answer']
    
    # Calculate ROUGE scores
    rouge_score = calculate_rouge(model_output, ground_truth)

Evaluation Metrics

• —ROUGE-1, ROUGE-2, ROUGE-L
• —Compression ratio accuracy
• —Extractive accuracy (sentences from original)

For training data, see: SahmBenchmark/financial-reports-extractive-summarization_train