h0ssn/wikitext-unlearning-mia

WIKITEXT - Machine Unlearning + MIA Evaluation Dataset (Length-Filtered)

This dataset is prepared for evaluating machine unlearning methods on fine-tuned LLMs using Membership Inference Attacks (MIAs).

Dataset Splits

Training Sets (for Unlearning)

• —`retain_set` (9,000 samples): Data to retain during unlearning
• —`forget_set` (1,000 samples): Data to unlearn

Evaluation Sets (for MIA) - Length-Filtered

WikiText Length Variants

• —96 tokens (~96±10 tokens): Medium passages
• —128 tokens (~128±10 tokens): Long passages (recommended)
• —192 tokens (~192±10 tokens): Very long passages

• —`retain_eval_96`, `forget_eval_96`: ~96 token versions
• —`retain_eval_128`, `forget_eval_128`: ~128 token versions (recommended)
• —`retain_eval_192`, `forget_eval_192`: ~192 token versions

Evaluation Dataset Structure

Each evaluation dataset (up to 700 samples) has the following structure:

Note: Actual sample counts may vary based on availability of samples matching the length criteria.

Length Filtering

All evaluation splits are filtered to match specific token lengths (±10 tokens tolerance). This length-matching prevents the model from using sequence length as a signal for membership inference, following the methodology from the Win-k MIA paper.

Usage

from datasets import load_dataset

# Load all splits
dataset = load_dataset("h0ssn/wikitext-unlearning-mia")

# Access training sets
retain_set = dataset["retain_set"]
forget_set = dataset["forget_set"]

# Access evaluation sets by length (use largest available for best results)
# These split names vary by dataset - check available splits
print(dataset.keys())

Use Case

This dataset is designed for:

1. 1.Machine Unlearning: Train models to "forget" the forgetset while retaining performance on retainset
2. 2.MIA Evaluation: Measure membership leakage before/after unlearning using the evaluation splits
3. 3.Privacy Research: Study the effectiveness of unlearning methods in protecting data privacy
4. 4.Length-Controlled MIA: Evaluate MIA attacks without length-based confounding factors

Citation

If you use this dataset, please cite the Win-k MIA paper which established the length-filtering methodology.

License

Please refer to the original dataset's license.