mehmettozlu/clean-tr-dataset

Turkish Clean Text Corpus

Dataset Description

This dataset is a highly cleaned, deduplicated, and normalized corpus of Turkish text. It features a diverse collection of informative, encyclopedic, and journalistic content.

The dataset is primarily designed for Continued Pre-Training (CPT) of Large Language Models (LLMs) to enhance their Turkish language capabilities and domain knowledge. It is also an excellent foundational corpus for generating synthetic Instruction/ChatML datasets or building large-scale Retrieval-Augmented Generation (RAG) systems.

• —Language: Turkish (tr)
• —Total Rows: 540,735
• —Average Word Count per Row: ~213 words
• —Format: Hugging Face Dataset (Arrow/Parquet)

Usage

You can easily download and use this dataset in your Python projects using the Hugging Face datasets library.

First, ensure you have the required library installed:

pip install datasets

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("mehmettozlu/clean-tr-dataset", split="train")

# Check the total number of rows
print(f"Total rows: {len(dataset)}")

# Print the first text example
print(dataset[0]["text"])

# Optional: Convert to Pandas DataFrame for easy manipulation
df = dataset.to_pandas()
print(df.head())

Dataset Structure

The dataset contains a single feature:

• —text (string): The cleaned, deduplicated Turkish text.

Example

{
  "text": "Ophrys apifera, Salepgiller ailesine bağlı, arı salebi olarak bilinen, bir orkide türüdür.\n\nBilimsel Tanımı\nOphrys apifera 20 ila 70 cm boylanabilen bir türdür. Gövde yapraksıdır. Çiçek kurulu 3 ila 8 arası çiçek barındırır. Çanak yapraklar iri, menekşe, gül ya da beyazımsı renklidir ve ortada yeşil bir damar görülür. Taç yapraklar yeşilimsi ya da morumsudur. Mayıs ve Haziran aylarında çayırlar..."
}