datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
codeparrot-valid-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (validation)
Dataset Description
A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-valid-near-deduplication.codeparrot-train-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (train)
Dataset Description
A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more details… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-train-near-deduplication.2505-deduplication-evals
HPLT 3.0: Deduplication Strategy Comparison Results
Dataset Description
This dataset contains fine-grained results from our HPLT 3.0 pre-release evaluations comparing different data deduplication stategies for the pre-HPLT 3.0 corpora with the previous HPLT 2.0 version. We compare the following data deduplication strategies to guide our design choices, and guard against data quality regression compared to HPLT 2.0: pre-HPLT 3.0 CD (per-crawl deduplication), pre-HPLT 3.0… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/2505-deduplication-evals.