AbrarHyder/Modified_german_dpr_dataset

Original Dataset

The original dataset, `deepset/germandpr`, contains:

• —9275 training examples
• —1025 testing examples

Each example is a question/answer pair, consisting of:

• —One question
• —One answer
• —One positive context
• —Three negative contexts

You can find the original dataset here.

Modifications

Adding Easy Negative Examples

To enhance the dataset, an "easy negative example" was added to each row. The objective of this addition is to train the model to better distinguish between relevant and irrelevant contexts by exposing it to plausible but incorrect information.

After adding easy neagtives each example in the dataset consists of :

• —One question
• —One answer
• —One positive context
• —Three negative contexts
• —Easy negative contexts

Method

For identifying easy negative examples, I have used utilized the L2 distance metric from Faiss to find the most dissimilar index (vector) relative to the positive context in each row. This dissimilar index was then selected as the easy negative example.