BSC-LT/bsc-dolly-15k-en

BSC Dolly 15k EN

Reviewed version from the Argilla Dolly v2 English version, originally created by Databricks.

We provide two subsets: "annotated", where some instances were labelled with potential problems; and "filtered", which only contains the instances without the issues that we observed.

Annotation process

While analysing the Argilla Dolly v2 English version, we observed the following:

1. 1.Task classification:
2. 2.There are three classes with context: 'Closed QA', 'Information Extraction' and 'Summarization'. The rest without context.
3. 3.Context is not necessary in all cases and there are instructions that already contain context.
4. 4.Incorrect categories (the intention does not always correspond to the category).
5. 5.
6. 6.Confusion between "Summarization" and "Open Generative QA" / "Information Extraction" tasks:
7. 7.Tasks categorized as "Summarization" have in some cases the intent of "Open Generative QA" / "Information Extraction", and due to their dependency on context, the answer is longer.

1. 1.To note:
2. 2.15,014 examples, half of "QA" type in various formats.
3. 3.70% have no context; when they do, they come from the first part of Wikipedia.
4. 4.Many answers are also from Wikipedia.
5. 5.Possible improvements in cleaning up text extracted from Wikipedia and handling acronyms.

1. 1.Errors in the dataset:
2. 2.Some summaries are longer than the original text.
3. 3.Some contexts in "Information Extraction" do not contain the exact information to answer the question asked.
4. 4.There are many repeated questions that are kept because the answer is different in each case.

From the previous observations, we performed the following processing:

• —Processed "context" column to:
• —Remove spellings, citations, or unit conversions inside (parenthesis) and [brackets].
• —Removed source webpage links.

• —Removed:
• —Summary instances where intent is clear & response is longer than context (63)
• —Instances where the information is not explicitly mentioned in the context (3)
• —Instances with webpage links in the response or instruction (29)
• —Exact (instruction/context/response) duplicates (14)
• —Instruction/context duplicates (9)
• —Instances where instruction is most similar to the response (6)
• —
• —Changes:
• —Some instances in Summarization/Information Extraction/ Closed QA are lacking context after Argilla's curation process. These instances are moved to General QA since they have no longer context and ask about specifics (86).