CoolFace
Datasetpublic

ongsici/Dolma-Paloma

Dataset Card for Dolma-Paloma Dataset Summary This dataset was created as part of my Master's thesis research on "Leveraging Model Checkpoints for Membership Inference Attacks on Large Language Models". The aim was to create a clean setup, free from distribution shifts, to study proposed checkpoint MIA methods and their performance using checkpoints from Pythia models. It was derived from the original Dolma and Paloma datasets by applying reservoir sampling and… See the full description on the dataset page: https://huggingface.co/datasets/ongsici/Dolma-Paloma.

sourceHugging Faceupdated 1y agoView on Hugging Face
0likes42downloads
Dataset Card

Dataset Card for Dolma-Paloma

Dataset Summary

This dataset was created as part of my Master's thesis research on "Leveraging Model Checkpoints for Membership Inference Attacks on Large Language Models". The aim was to create a clean setup, free from distribution shifts, to study proposed checkpoint MIA methods and their performance using checkpoints from Pythia models. It was derived from the original Dolma and Paloma datasets by applying reservoir sampling and data processing steps.

Data Processing

Random sampling was applied to Paloma (Dolma v1.5 domain) dataset to obtain domains with >100 non-members from val and test splits that with sequence lengths >=2048. Reservoir sampling was applied to Dolma v1.5 to obtain >100 members for the same domains from train split with sequence lengths >=2048.

Samples were tokenised using Llama-2-7B model and samples with above 2048 token lengths were retained. Sample tokens were then truncated to various sequence lengths: [64, 128, 256, 512, 1024, 2048], and the sample sequences were then regenerated from the tokens.

This was applied to 3 domains: Wikipedia, Stack, and Pes2o.

Dataset Information

Column NameDescription
textThe raw text sample from the dataset.
subdomainThe domain from Dolma/Paloma that the sample belongs to.
llama_input_idsTokens obtained after tokenising the sample using Llama-2-7B tokenizer.
llama_attention_maskAttention mask corresponding to llama_input_ids, indicating valid tokens.
seq_lenSequence lengths measured by number of tokens from llama_input_ids
labelMembership label (1 = member, 0 = non-member).
truncated_textText regenerated from llama_input_ids after truncating to the desired sequence lengths

Intended Use

The dataset is intended for academic research purposes, particularly for evaluating methods for MIAs.