h0ssn/agnews-unlearning-mia

AGNEWS - Machine Unlearning + MIA Evaluation Dataset (Length-Filtered)

This dataset is prepared for evaluating machine unlearning methods on fine-tuned LLMs using Membership Inference Attacks (MIAs).

Dataset Splits

Training Sets (for Unlearning)

• —`retain_set` (9,000 samples): Data to retain during unlearning
• —`forget_set` (1,000 samples): Data to unlearn

Evaluation Sets (for MIA) - Length-Filtered

AGNews Length Variants

• —32 tokens (~32±10 tokens): Short headlines
• —48 tokens (~48±10 tokens): Medium headlines
• —64 tokens (~64±10 tokens): Long headlines

• —`retain_eval_32`, `forget_eval_32`: ~32 token versions
• —`retain_eval_48`, `forget_eval_48`: ~48 token versions
• —`retain_eval_64`, `forget_eval_64`: ~64 token versions

Evaluation Dataset Structure

Each evaluation dataset (up to 700 samples) has the following structure:

Note: Actual sample counts may vary based on availability of samples matching the length criteria.

Length Filtering

All evaluation splits are filtered to match specific token lengths (±10 tokens tolerance). This length-matching prevents the model from using sequence length as a signal for membership inference, following the methodology from the Win-k MIA paper.

Usage

from datasets import load_dataset

# Load all splits
dataset = load_dataset("h0ssn/agnews-unlearning-mia")

# Access training sets
retain_set = dataset["retain_set"]
forget_set = dataset["forget_set"]

# Access evaluation sets by length (use largest available for best results)
# These split names vary by dataset - check available splits
print(dataset.keys())

Use Case

This dataset is designed for:

1. 1.Machine Unlearning: Train models to "forget" the forgetset while retaining performance on retainset
2. 2.MIA Evaluation: Measure membership leakage before/after unlearning using the evaluation splits
3. 3.Privacy Research: Study the effectiveness of unlearning methods in protecting data privacy
4. 4.Length-Controlled MIA: Evaluate MIA attacks without length-based confounding factors

Citation

If you use this dataset, please cite the Win-k MIA paper which established the length-filtering methodology.

License

Please refer to the original dataset's license.