huglabs/cemig-normas-v2-judged-50-gpt-oss-120b

CEMIG Distribution Standards Grounded QA Benchmark

Dataset summary

This dataset contains 50 synthetic, multi-context question-answer pairs grounded in publicly classified CEMIG technical distribution standards. It was created to evaluate retrieval-augmented generation (RAG) and grounded question answering in the electrical-distribution domain. Each question and reference answer is in English and is associated with two Portuguese source passages, retrieval similarities, generation metadata, an automatic quality judgement, and an item-specific coverage checklist.

The benchmark is intended for research and comparative evaluation. It is not an official CEMIG publication, does not replace the source standards, and must not be used as operational, engineering, safety, legal, or regulatory guidance.

Source corpus

The source corpus is the Hugging Face dataset `cemig-ceia/normas_tecnicas_distribuicao_markdown`, using the train split pinned to revision `43ef9a077a3cd138b483dedbf232b98eb775041e`. That revision contains 32 Markdown-extracted technical documents. The 50 benchmark examples use 96 distinct chunks from 21 source documents.

The source repository does not currently declare a Dataset Card or license. Users must independently verify the applicable terms for the source documents and extracted passages before redistributing or using the data. The documents remain authoritative only in their official, current versions.

Languages

• —Source documents and retrieved contexts: Portuguese (pt).
• —Questions and reference answers: English (en).
• —Automatic judge explanations: English.
• —Coverage checklist items: English, preserving the question language.

The metadata field language: en in the JSON records the configured generation language. It does not mean that the source contexts are English.

Dataset generation

Pipeline parameters

Pipeline description

1. 1.The pinned source corpus was read from Hugging Face and deduplicated using a prefer_longest policy.
2. 2.Portuguese Markdown documents were divided into semantic chunks. Chunking targeted 500 tokens, accepted chunks between 100 and 1,000 tokens, preserved undersized units, and allowed merging across sections.
3. 3.Chunks were embedded with Qwen/Qwen3-Embedding-0.6B and stored in the cemig_normas_qwen3_0_6b_v1 vector collection.
4. 4.Two chunks were selected for each candidate using the configured random strategy and seed. Up to 10 context-selection attempts were allowed.
5. 5.openai.gpt-oss-120b received the selected passages and generated one integrated English question and its grounded reference answer using prompt strategy grounded_qa, version 2, at temperature 0.2.
6. 6.Each candidate was evaluated by the automatic judge before being retained. Only approved candidates received an ID and entered the benchmark.
7. 7.Query-to-context and context-to-context cosine similarities were calculated and stored with each accepted example.
8. 8.A separate stage generated and semantically reviewed a coverage checklist for every accepted QA pair.

The reproducible local configurations are benchmark_generation/configs/cemig_normas_v2_judged_50_gpt_oss_120b.yaml and benchmark_generation/configs/cemig_normas_v2_coverage_checklists_50_gpt_oss_120b.yaml. Infrastructure-specific endpoint addresses and credentials are intentionally not included in this card.

Automatic quality judgement

Judge parameters

Judge procedure

The V3 judge evaluated every candidate against five independent criteria. An approved question must not refer directly to the supplied passages, must require substantive information from both passages, must be a single integrated request, must meaningfully relate both passages, and must not disclose the evidence or reasoning expected in the answer.

The judge assessed structural validity for this benchmark design. Its rubric did not independently certify the prose quality or factual correctness of each reference answer. Generation and judgement used the same model family, which may introduce correlated errors and self-consistency bias.

Coverage checklists

Each example includes an item-specific coverage_checklist. It represents the explicit answer requirements found in the question, with the reference answer used only to fill in the expected result. Checklist items are intended as atomic, non-overlapping binary tests and accept semantically equivalent wording.

Dataset statistics

The 50 retained rows are approved outputs, not an acceptance-rate report. The artifact does not preserve every rejected candidate, so it must not be interpreted as showing a 100% candidate approval rate.

Data structure

The distributed JSON has two top-level fields: metadata and qas. Each object inside qas contains:

Loading the file

Because the rows are nested under the qas key, load the JSON explicitly:

import json
from pathlib import Path

path = Path("cemig_normas_v2_judged_50_gpt_oss_120b.json")
payload = json.loads(path.read_text(encoding="utf-8"))
metadata = payload["metadata"]
qas = payload["qas"]

With the datasets library and a downloaded local copy:

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files="cemig_normas_v2_judged_50_gpt_oss_120b.json",
    field="qas",
    split="train",
)

Intended uses

• —Evaluate context-grounded question answering.
• —Compare RAG systems in the electrical-distribution domain.
• —Evaluate retrieval over Portuguese technical passages with English queries.
• —Measure coverage of explicit answer requirements.
• —Study multi-context synthesis and answer grounding.

Out-of-scope uses

• —Replacing current official CEMIG standards.
• —Producing operational engineering or electrical-safety instructions.
• —Demonstrating regulatory or contractual compliance.
• —Treating the generated reference answer as the only valid answer.
• —Evaluating unrelated domains or general-purpose factual knowledge.

Limitations and risks

• —The dataset is small and contains only 50 examples.
• —Questions, reference answers, judgements, and coverage checklists are synthetic.
• —No specialist human validation is recorded.
• —The same model generated and judged the QA pairs.
• —Automatic approval does not guarantee factual correctness or completeness.
• —Results depend on the corpus revision, semantic chunking, context selection, embedding model, and prompts used.
• —Cross-language retrieval from English questions to Portuguese contexts may introduce language-dependent effects.
• —Source documents may become outdated; consult the current official standards.
• —Long generated reference answers may reward verbosity in some evaluation methods.
• —Licensing and redistribution terms for the source corpus are not declared in its Hugging Face repository and require verification.

Versioning

• —dataset_version: 1.2 identifies the judged dataset schema produced by the benchmark generator.
• —v2 in the artifact name identifies generation prompt version 2; it is not a conflicting semantic-version number.
• —Source corpus revision: 43ef9a077a3cd138b483dedbf232b98eb775041e.
• —The original generation date was not recorded in the artifact. It is therefore intentionally not inferred from filesystem timestamps.

License and attribution

No license is declared in this card because the pinned source dataset currently does not state one. A license should be added only after confirming the rights applicable to the source passages and the generated derivative dataset.

When using this benchmark, attribute the source corpus to CEMIG-CEIA and link to the pinned Hugging Face dataset. A formal citation entry can be added when the publisher supplies the desired authorship, title, year, and repository identifier.