datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
codeparrot-valid-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (validation)
Dataset Description
A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-valid-near-deduplication.codeparrot-train-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (train)
Dataset Description
A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more details… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-train-near-deduplication.KingDesign-Deduplication
Kingdesign
Made with ❤️ using 🦥 Unsloth Studio
all was generated with Unsloth Recipe Studio. It contains 5,351 generated records.
🚀 Quick Start
from datasets import load_dataset
# Load the main dataset
dataset = load_dataset("JaveyZou/KingDesign", "data", split="train")
df = dataset.to_pandas()
📊 Dataset Summary
📈 Records: 5,351
📋 Columns: 3
✅ Completion: 89.2% (6,000 requested)
📋 Schema & Statistics
Column
Type
Column Type
Unique (%)… See the full description on the dataset page: https://huggingface.co/datasets/JaveyZou/KingDesign-Deduplication.2505-deduplication-evals
HPLT 3.0: Deduplication Strategy Comparison Results
Dataset Description
This dataset contains fine-grained results from our HPLT 3.0 pre-release evaluations comparing different data deduplication stategies for the pre-HPLT 3.0 corpora with the previous HPLT 2.0 version. We compare the following data deduplication strategies to guide our design choices, and guard against data quality regression compared to HPLT 2.0: pre-HPLT 3.0 CD (per-crawl deduplication), pre-HPLT 3.0… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/2505-deduplication-evals.exp_1_4_deduplication_deduplicatedtest-deduplication-demotext-based-deduplication-longtext-based-deduplication