ymoslem/news-commentary-en-ar

This is a filtered version of the English-to-Arabic News Commentary dataset available at data.statmt.org/news-commentary. The filtering process includes removing duplicates, language detection, and semantic filtering based on similarity (>0.70) between the source and translation. The filtering script is available at data-processing.ipynb.