renataaraujoe/Bilingual-LLM-Eval-106

Bilingual-LLM-Eval-106

📌 Overview

Bilingual-LLM-Eval-106 is a curated, human-annotated evaluation dataset of 106 LLM response pairs in English and Portuguese, designed to benchmark model performance across multiple quality dimensions.

The dataset focuses on realistic and challenging evaluation scenarios, including adversarial prompts, ambiguous queries, and hard negatives.

It is intended for:

• —LLM evaluation and benchmarking
• —LLM-as-a-Judge research
• —Multilingual robustness analysis
• —Annotation quality research

🎯 Objectives

This dataset was built to:

• —Design and apply a professional annotation rubric
• —Collect diverse prompt types:
• —Factual
• —Reasoning
• —Adversarial
• —Ambiguous
• —Annotate responses using multi-dimensional quality labels
• —Ensure bilingual parity (EN + PT) with equivalent difficulty
• —Validate annotation consistency using:
• —Cohen’s Kappa
• —Bilingual calibration
• —Study LLM-as-a-Judge biases through controlled experiments

📊 Dataset Structure

The dataset consists of annotated CSV files:

• —annotations_EN_batch1.csv → English samples
• —annotations_PT_batch1.csv → Portuguese samples
• —annotations_mirrored_batch.csv → Cross-lingual mirrored examples
• —annotations_edge_cases.csv → Adversarial and difficult cases

Each row represents a prompt + response pair with annotations.

🧾 Annotation Dimensions

Each response is evaluated across multiple dimensions:

• —Faithfulness → factual correctness and grounding
• —Relevance → alignment with the prompt
• —Fluency → linguistic quality
• —Completeness → coverage of required information
• —Safety → harmful or risky content

Labels follow a structured rubric inspired by industry standards used in:

• —Scale AI
• —Anthropic
• —DataAnnotation

🌍 Languages

• —English (EN)
• —Portuguese (PT)

The dataset includes:

• —Independent annotations per language
• —Mirrored examples for cross-lingual consistency analysis

🧪 Research Use Cases

This dataset enables:

🔍 LLM Evaluation

Benchmark models across multiple qualitative dimensions beyond accuracy.

⚖️ LLM-as-a-Judge Analysis

Study bias, inconsistency, and failure modes in automated evaluation systems.

🌐 Multilingual Testing

Compare performance across English and Portuguese under equivalent conditions.

🎯 Robustness Testing

Evaluate models on:

• —Edge cases
• —Adversarial prompts
• —Ambiguous inputs

📏 Annotation Quality

• —Annotated following a custom-built professional rubric
• —Includes hard negatives and adversarial cases
• —Validated with:
• —Inter-annotator agreement (Cohen’s Kappa)
• —Cross-lingual calibration

📁 Data Format

Typical columns may include:

• —prompt
• —response
• —language
• —faithfulness
• —relevance
• —fluency
• —completeness
• —safety
• —notes (optional)

⚠️ Limitations

• —Dataset size is intentionally small (106 samples) for high-quality evaluation, not training
• —Domain coverage is diverse but not exhaustive
• —Some annotations may include subjective judgment despite calibration

🤝 Contributions

This dataset was created as an independent research and engineering project focused on LLM evaluation quality and methodology.

⭐ Citation

@dataset{bilingualllmeval1062026, author = {Renata de Araujo}, title = {Bilingual-LLM-Eval-106: A Human-Annotated Benchmark for English–Portuguese LLM Evaluation}, year = {2026}, publisher = {renataaraujoe}, howpublished = {\url{https://huggingface.co/datasets/renataaraujoe/Bilingual-LLM-Eval-106}}, note = {Version 1.0} }