boyiwei/CoTaEval

CoTaEval Dataset

CoTaEval Dataset is used to evaluate the feasibility and the side effects of copyright takedown methods for language models. The dataset has two domains: News and Books. For News, it has three subsets: `news_for_unlearning (for unlearning use), news_infringement(for infringement evaluation), and news_utility (for utility evaluation); For Books, it has two subsets: books_infringement (for infringement evaluation), books_utility` (for utility evaluation). The structure of the dataset is shown below:

• —CoTaEval
• —News
• —newsforunlearning
• —forget_set (1k rows)
• —retain_set (1k rows)
• —news_infringement
• —blocklisted (1k rows)
• —news_utility
• —blocklisted (500 rows)
• —in_domain (500 rows)
• —Books
• —books_infringement
• —blocklisted (500 rows)
• —books_utility
• —blocklisted (500 rows, here we only use first 200 rows for utility evaluation)
• —in_domain (200 rows)

Usage

For infringement test (take news as an example):

from datasets import load_dataset

dataset = load_dataset("boyiwei/CoTaEval", "news_infringement", split="blocklisted")

We use `prompt_autocomplete as hint to prompt the model, and compute 8 infringement metrics between the generated content and gt_autocomplete`.

For utility test (take news as an example):

from datasets import load_dataset

dataset = load_dataset("boyiwei/CoTaEval", "news_utility", split="blocklisted") # use split="in_domain" for in-domain utility test

For news, we use `question to prompt the model, and compute the F1 score between the generated content and answer. For books, we ask the model to summarize the books chapter and compte the ROUGE score between the generated content and summary`.

For unlearning (please refer to TOFU for more details)

from datasets import load_dataset

dataset = load_dataset("boyiwei/CoTaEval", "news_for_unlearning", split="forget_set") # use split="retain_set" to get retain set