anonymous-aardvark/submission14717_fictionalqa_training_splits

Training splits view of the FictionalQA dataset

The FictionalQA dataset

• —Repository: omitted
• —Paper: omitted

Dataset Description

This dataset is a derivative of the main dataset. Please see that dataset's README for a detailed description of the assets.

The dataset splits (configs) provided here are the exact ones materialized and used in the experiments for the associated paper. The primary purpose of this dataset repository is for transparency and to help understand the experimental results in the paper. As such, the names of the configs are extremely verbose in service of being explicit and self-describing. Please see the experimental section of the paper to understand the splitting process used. The names of certain columns such as text, input, and response are chosen to align with the column names commonly expected in LLM training codebases for pretraining on webtext and finetuning on instruction and response pairs.

One notable inclusion in this repository is the multiple choice formatted versions of the fictional Q&A pairs presented in the main dataset. The approach used to construct this alternate answers for multiple choice evaluation was a post-hoc process and is thus part of the derived view of the data. Future work could more natively integrate the multiple choice formatting into the question generation stage of the pipeline. Please see the relevant section in the paper for a description of the multiple choice construction process.

Supported Tasks

This dataset supports language model training experiments of various kinds. The fiction documents and structured fictsheets can both be used as plain-text documents in a pretraining setting and the question and answer pairs can be used as instruction and response pairs for finetuning style experiments. However, because of their synthetic, fictional nature, measurements of language modeling performance and question answering performance on this data are largely uninfluenced by other training data making it ideal for studying memorization, knowledge acquisition, unlearning, and many other topics, in a controlled manner even on top of pretrained based models.

Multiple choice question evals

As part of the experiments, we utilize the *mcq* configs provided in this dataset as source data for use in Eleuther's lm-eval-harness. The dir containing the task definitions required to run the MCQ tests in the harness is provided as a set of yaml files at the realtive path lm_eval/tasks/fictional_qa in the generation repo (linked above) and needs to be copied into a copy of the lm-eval-harness repository in order to be run.

Citation

omitted