nguyenkhanh87/ViLegalQA-Synthetic-Curation

ViLegalQA Synthetic Curation

Dataset summary

This repository releases the synthetic Vietnamese legal QA research artifacts produced in the accompanying study. The primary resource contains 10,095 synthetic QA items spanning true/false, multiple-choice, and open-ended tasks. It is accompanied by the final curation/quality annotations used in the study, plus aggregated labels for 600 items from the five-expert human calibration panel.

Manuscript: Human-Calibrated Reliability-Weighted LLM Juries for Risk-Budgeted Synthetic Legal QA Curation (URL to be added).

Copyright-conscious release design

The synthetic items were generated from legal source material in the ALQAC 2025 Law Corpus. This repository does not redistribute the ALQAC source-law context text or the locked ALQAC benchmark. Instead, each item stores only the source identifiers using the ALQAC-compatible structure:

"relevant_articles": [
  {
    "law_id": "Luật Hôn nhân và gia đình",
    "article_id": "3"
  }
]

Users who require the original legal context should obtain ALQAC through its official distribution channel and comply with its access/licensing terms: https://huggingface.co/datasets/nguyenlab/ALQAC

Each synthetic item in this release was generated from one source article, so relevant_articles currently contains one element per item. The field is intentionally represented as a list to remain compatible with the ALQAC data model and future multi-article extensions.

Configurations

synthetic_qa (default)

ALQAC-style synthetic QA records:

• —question_id: synthetic item identifier.
• —question_type: one of Đúng/Sai, Trắc nghiệm, or Tự luận.
• —text: generated question.
• —choices: mapping from option labels to option text for MCQ items; absent otherwise.
• —relevant_articles: list of {law_id, article_id} references.
• —answer: synthetic answer associated with the generated item; it should not be interpreted as expert-validated gold unless supported by the human calibration annotations.

quality_annotations

Final study-level quality and curation metadata keyed by question_id, including the reliability-weighted item score, task-specific threshold, criterion-level weighted scores and evidence counts, and selection indicators for Majority-3, Majority-4, Unanimity-5, and reliability-weighted curation.

human_calibration

Aggregated outputs of the five-expert calibration panel. This configuration contains final panel labels and aggregate vote/support information only; no expert names, emails, or other identity fields are released.

jury_replay

Sanitized per-item LLM-jury decisions used for public statistical and selection replay. The configuration excludes ALQAC source-law text, question/answer text, prompts, raw model completions, and token-level generation metadata. It includes the criterion-level judge decisions required by Steps 07--09 and the derived jury provenance fields primary_pattern, positive_count, and semantic_bucket required to reproduce the Step-09 selection manifest.

Selector metadata

The metadata/ directory contains small study-level artifacts such as task-specific judge reliability estimates, calibrated operating points, selector metrics, and the risk--coverage curve when these files are available in the project.

Intended use

This release is intended for research on Vietnamese legal QA, synthetic-data quality assessment, LLM-as-a-Judge aggregation, data curation, and alternative selection strategies. Curation indicators should not be interpreted as universal rankings of training usefulness. The accompanying study finds that lower estimated validity risk can coexist with worse downstream utility when filtering changes the retained training distribution.

Limitations

• —The questions and answers are synthetic rather than expert-authored legal advice.
• —Human calibration covers a subset of the synthetic pool.
• —The current generated items are grounded in a single source article even though relevant_articles supports multiple references.
• —Original ALQAC source text is intentionally excluded from this release.
• —Automated quality scores are study-specific measurements and should not be treated as legal correctness guarantees.

Loading with Hugging Face Datasets

from datasets import load_dataset

repo = "nguyenkhanh87/ViLegalQA-Synthetic-Curation"

qa = load_dataset(repo, "synthetic_qa")
quality = load_dataset(repo, "quality_annotations")
human = load_dataset(repo, "human_calibration")
jury = load_dataset(repo, "jury_replay")

Citation

Please cite the accompanying manuscript. A complete BibTeX entry should be added here once the manuscript metadata/DOI or preprint identifier is finalized.

Release integrity

release_manifest.json records row counts, input artifact hashes, and design exclusions. SHA256SUMS.txt contains hashes for the public release files.