azhx/counterfact-filtered-gptj6b
Dataset Card for "counterfact-filtered-gptj6b" This dataset is a subset of azhx/counterfact-easy, however it was filtered based on a heuristic that was used to determine whether the knowledge in each row is actually known by the GPT-J-6B model The heuristic is as follows: For each prompt in the original counterfact dataset used by ROME, we use GPT-J-6B to generate n=5 completions to a max generated token length of 30. If the completion contains the answer that is… See the full description on the dataset page: https://huggingface.co/datasets/azhx/counterfact-filtered-gptj6b.
Dataset Card for "counterfact-filtered-gptj6b"
This dataset is a subset of azhx/counterfact-easy, however it was filtered based on a heuristic that was used to determine whether the knowledge in each row is actually known by the GPT-J-6B model
The heuristic is as follows:
For each prompt in the original counterfact dataset used by ROME, we use GPT-J-6B to generate n=5 completions to a max generated token length of 30. If the completion contains the answer that is specified in the dataset for the majority of the completions (>=3), then we conclude that the model does indeed know this fact.
In practice, we find that many of the prompts in the original dataset cannot be answered accurately a lot of the time using GPT-J-6B. The number of case_ids were filtered from ~21k to about ~3k.