fyaronskiy/ru-paraphrase-NMT-Leipzig-cleaned

Dataset Description

The dataset is obtained by filtering dataset of russian paraphrases by David Dale with automatic metrics. The data structure is saved.

Have been deleted:

• —Paraphrases that have cosine LABSE similarity with source sentences < 0.75.
• —Paraphrases that are more than 2.5 times longer than source sentences. (Most of them are looped errors of back translation)
• —Paraphrases that are similar in spelling to the original texts (paraphrases that have ChrF++ similarity > 0.6 and consist of 15 or more tokens) have been deleted.

Source dataset contained samples with English paraphrases (errors of back translation). This paraphrases have been translated into Russian.

Supported Tasks and Leaderboards

The dataset can be used to

• —train and evaluate models for paraphrase generation or (if negative sampling is used) for paraphrase detection.
• —train sentence embeddings models.