deduplication
codeparrot-valid-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (validation)
Dataset Description
A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-valid-near-deduplication.codeparrot-train-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (train)
Dataset Description
A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more details… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-train-near-deduplication.core-2020-05-10-deduplication
Dataset Card for CORE Deduplication
Dataset Summary
CORE 2020 Deduplication dataset (https://core.ac.uk/documentation/dataset) contains 100K scholarly documents labeled as duplicates/non-duplicates.
Languages
The dataset language is English (BCP-47 en)
Citation Information
@inproceedings{dedup2020,
title={Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings},
author={Gyawali, Bikash and Anastasiou, Lucas and… See the full description on the dataset page: https://huggingface.co/datasets/pinecone/core-2020-05-10-deduplication.KingDesign-Deduplication
Kingdesign
Made with ❤️ using 🦥 Unsloth Studio
all was generated with Unsloth Recipe Studio. It contains 5,351 generated records.
🚀 Quick Start
from datasets import load_dataset
# Load the main dataset
dataset = load_dataset("JaveyZou/KingDesign", "data", split="train")
df = dataset.to_pandas()
📊 Dataset Summary
📈 Records: 5,351
📋 Columns: 3
✅ Completion: 89.2% (6,000 requested)
📋 Schema & Statistics
Column
Type
Column Type
Unique (%)… See the full description on the dataset page: https://huggingface.co/datasets/JaveyZou/KingDesign-Deduplication.2505-deduplication-evals
HPLT 3.0: Deduplication Strategy Comparison Results
Dataset Description
This dataset contains fine-grained results from our HPLT 3.0 pre-release evaluations comparing different data deduplication stategies for the pre-HPLT 3.0 corpora with the previous HPLT 2.0 version. We compare the following data deduplication strategies to guide our design choices, and guard against data quality regression compared to HPLT 2.0: pre-HPLT 3.0 CD (per-crawl deduplication), pre-HPLT 3.0… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/2505-deduplication-evals.deduplication
Semantic Deduplication UV Script
Part of uv-scripts — self-contained UV scripts you run locally or on Hugging Face Jobs in one command.
Remove duplicate / near-duplicate text samples from a Hugging Face dataset by semantic similarity — clean training data and prevent train/test leakage. Uses SemHash with Model2Vec embeddings: CPU-optimized, no GPU required.
Quick start
# CPU is enough — run on Hugging Face Jobs
hf jobs uv run --flavor cpu-upgrade --secrets… See the full description on the dataset page: https://huggingface.co/datasets/uv-scripts/deduplication.