Datasetpublic

Neural-Navigator-Labs/wmt14_en-de

SmolTrans EN-DE Dataset High-quality English–German parallel dataset filtered using: Language detection HTML removal Duplicate removal LM perplexity filtering Perplexity ratio filtering Embedding cosine similarity Statistics Total pairs: 4.5M Median length: ... Filtering removed: ~X% Format {"en": "...", "de": "..."} Usage from datasets import load_dataset ds = load_dataset("/smoltrans-en-de") Notes Short sentences… See the full description on the dataset page: https://huggingface.co/datasets/Neural-Navigator-Labs/wmt14_en-de.

sourceHugging Faceupdated 6mo agoView on Hugging Face

0likes13downloads

Dataset Card

SmolTrans EN-DE Dataset

High-quality English–German parallel dataset filtered using:

—Language detection
—HTML removal
—Duplicate removal
—LM perplexity filtering
—Perplexity ratio filtering
—Embedding cosine similarity

Statistics

—Total pairs: 4.5M
—Median length: ...
—Filtering removed: ~X%

Format

{"en": "...", "de": "..."}

Usage

from datasets import loaddataset ds = loaddataset("<your-username>/smoltrans-en-de")

Notes

Short sentences may show unstable perplexity values.