MarcGrumpyOlejak/gooaq_mt_german_0_hard_negatives

Remaining GooAQ (Google Answers to Google Questions) question-answer pairs in German without hard negatives.

About

This dataset contains the remaining 600K of lines of german machine translated texts of the mined hard negatives ~2M question-answer-negative triplets and question-answer-negative1...-negative5 tuples gooaq_mt_german_5_hard_negatives. The full original Gooaq dataset in english only: (link to original dataset). This dataset can be used directly with Sentence Transformers to train embedding models.

I'll suggest to use it in combination with the gooaq_mt_german_5_hard_negatives dataset.

Dropped lines without hard negatives

The remaining 623_000 deduplicated lines of material here are so unique that no negatives could get mined before and they have been rescued.

'Remaining' sentences are mainly due 2 reasons to 'fall' out of the mined results table:

• —The quality of the anchor/query is so low in combination with the positives/answers.
• —or the quality of the anchor is so unique and good but is still not a part of the mining model.

(I have to check out how to combine datasets into subsets - yet not done before)

Usage

Using the HuggingFace datasets library:

from datasets import load_dataset
dataset = load_dataset('MarcGrumpyOlejak/gooaq_mt_german_0_hard_negatives')['train']
print(dataset)

Dataset({
    features: ['question_mt_de', 'answer_mt_de'],
    num_rows: 623131
})

Suggested Train/Test split

dataset_dict = dataset.train_test_split(test_size=1_000, seed=12)
print(dataset_dict)
DatasetDict({
    train: Dataset({
        features: ['question_mt_de', 'answer_mt_de'],
        num_rows: 622131
    })
    test: Dataset({
        features: ['question_mt_de', 'answer_mt_de'],
        num_rows: 1000
    })
})