hallisky/wikiMIA-2024-hard

WikiMIA-2024 Hard Dataset

Dataset Description

WikiMIA_2024 Hard is a challenging dataset for membership inference attacks intorduced in the paper "The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage" containing temporal Wikipedia articles with different versions based on date cutoffs. This dataset is designed to evaluate the robustness of privacy-preserving machine learning models against sophisticated membership inference techniques.

It differs from WikiMIA in that 1) It uses updated non-member cutoffs of 2024, enabling attacks on contemporary models like GPT-4 and 2) collects member/non-member pairs based on versions of the same article at different points in time, which reduces spurious signals in the original WikiMIA paper.

Citation

If you use/reference this work, please cite us via:

@inproceedings{hallinan2025surprising,
  title={The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage},
  author={Hallinan, Skyler and Jung, Jaehun and Sclar, Melanie and Lu, Ximing and Ravichander, Abhilasha and Ramnath, Sahana and Choi, Yejin and Karimireddy, Sai Praneeth and Mireshghallah, Niloofar and Ren, Xiang},
  booktitle={Conference on Language Modeling (CoLM)},
  year={2025}
}