CoolFace
Datasetpublic

Ba2han/en-tur_corpus

English - Turkish Filtered Corpus This dataset was compiled from multiple English and Turkish sources, heavily filtered, and deduplicated. Processing Applied: Length Filtering: Min characters = 100, Max characters = 2600 Deduplication: Exact deduplication followed by fast heuristic fingerprinting (normalized, whitespace & punctuation removed) Shuffled: Seed = 42 Dataset Composition: Dataset Source Original Rows (Loaded) Final Rows (After… See the full description on the dataset page: https://huggingface.co/datasets/Ba2han/en-tur_corpus.

sourceHugging Faceupdated 7mo agoView on Hugging Face
0likes230downloads
Dataset Card

English - Turkish Filtered Corpus

This dataset was compiled from multiple English and Turkish sources, heavily filtered, and deduplicated.

Processing Applied:

  1. 1.Length Filtering: Min characters = 100, Max characters = 2600
  2. 2.Deduplication: Exact deduplication followed by fast heuristic fingerprinting (normalized, whitespace & punctuation removed)
  3. 3.Shuffled: Seed = 42

Dataset Composition:

Dataset SourceOriginal Rows (Loaded)Final Rows (After Filters & Dedup)
jordiclive/wikipedia-summary-dataset7,750,0075,993,863
LocalDoc/news_azerbaijan447,197411,617
ccdv/pubmed-summarization119,924119,783
turkish-nlp-suite/temiz-Wiki360,175290,193
Ba2han/vngrs-web-filtered1,000,000183,604
argilla/cnn-dailymail-summaries287,113287,051
turkish-nlp-suite/Havadis744,868334,579
bigscience-data/rootsenthepileuspto1,000,000395,842
SocialGrep/one-million-reddit-jokes10,4496,650
karpathy/fineweb-edu-100b-shuffle1,000,000423,426
musabg/wikipedia-tr-summarization119,110118,550
common-pile/arxivabstractsfiltered2,504,6792,498,158
HuggingFaceFW/finetranslations1,000,000378,533
TOTAL16,343,52211,441,849