spadeMIA/GovReport_Corpus_1024_2040

GovReport MIA fine-tuning corpus

This card documents the split layout for `spadeMIA/GovReport_Corpus_1024_2040`, a GovReport corpus used for membership-inference attack (MIA) experiments on fine-tuned language models.

Membership labels are defined relative to the fine-tuning population. train is the only split used for fine-tuning, and every row has label = 1. test remains held out, and every row has label = 0. evaluation is a fixed, balanced MIA candidate set. Fine-tune on train only. Never use test or evaluation for gradient updates, early stopping, checkpoint selection, or attack hyperparameter tuning.

Project and team

This corpus was prepared for a membership inference attack research project at the Security, Privacy and Data Engineering (SPADE) Lab, Koç University.

Schema

All three splits use the same two-column schema:

label is benchmark metadata. It is not the token-level causal-LM labels tensor and must be removed before the data reach a language-model training collator.

Dataset layout and loading

This release intentionally declares no configs: stanza. Its top-level data/ directory permits automatic filename discovery of exactly three splits without a configuration or subset argument:

from datasets import load_dataset

repo_id = "spadeMIA/GovReport_Corpus_1024_2040"
train = load_dataset(repo_id, split="train")
test = load_dataset(repo_id, split="test")
evaluation = load_dataset(repo_id, split="evaluation")

Evaluation-set provenance and semantics

evaluation is the byte-identical benchmark artifact from `spadeMIA/govreport_evaluation_benchmark`, whose source artifact is mia_benchmark_v2.parquet at revision c64a97cc432822ad2445d12adce73a8a246ab85b. It is copied without row, text, label, order, or byte changes into data/evaluation-00000-of-00001.parquet.

All 350 evaluation rows with label = 1 occur in train. The 350 rows with label = 0 are independent of both current train and current test. They contain 348 unique exact text values because two text values are each duplicated once within the non-member class. This artifact is retained as-is, rather than regenerated or deduplicated during the schema migration.

Parent-leakage repair

The pre-repair source had 10,000 train rows but only 9,982 unique exact texts; two exact texts also appeared in the 1,000-row test split. The duplicate rows were identified deterministically and replaced in place with two records from the frozen ccdv/govreport-summarization validation split. This restores the locked 1,000-row test protocol without using any record from train or evaluation.

Removed original test rows:

Replacement rows:

The resulting test split has 1,000 rows, zero exact text overlap with train, and zero exact text overlap with evaluation. The deterministic repair and count restoration are implemented in Domains/GovReport/data/repair_govreport_parent_leakage.py and Domains/GovReport/data/restore_govreport_test_count.py; their pinned manifest is govreport_1024_2040_parent_leakage_repair.json in the dataset repository.

The evaluation artifact remains unchanged, including its two duplicate text values within the non-member class. This is an intra-class duplicate and does not create train/test membership-label leakage.

Migration invariants

The schema migration adds only the constant scalar label column to the existing train and repaired test rows. Train text and order are unchanged. Test row positions are unchanged, with only the two documented duplicate positions replaced by source-disjoint records. The evaluation Parquet artifact is copied byte-for-byte from its separate source repository and not reconstructed.

Ordered content hashes use SHA-256 over the length-prefixed UTF-8 text sequence:

train:       c1f413d37b8131636891b45a4e64bfeeb08b9ea28fde43254f11370ab8745c65
test:        554dc48273cf9c9f4ccdde179643c46da72f0bf48fe2a69b271d38199daefb19
evaluation:  d100b817eb53ba628dea60537e9835d42f1ec3bfe197245a2ed3dc942dfd90e9

The evaluation Parquet SHA-256 is:

5f333398a9ac8ff169005628a5b82504e543be90434557c22a3d2f29a865bcc5

Release validation checks all three schemas, split counts, label distributions, ordered text hashes, evaluation-byte identity with the separate benchmark repository, and load_dataset(repo_id, split=...) for every split name.

Source revisions

The migration was built from: