common-pile/data_provenance_initiative

Data Provenance Initiative Description The Data Provenance Initiative is a digital library of supervised datasets that have been manually annotated with their source and license information [ 104, 107 ]. We leverage their tooling to filter HuggingFace datasets, based on a range of criteria, including their licenses. Specifically, we filter the data according to these criteria: contains English language or code data, the text is not model-generated, the dataset’s… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/data_provenance_initiative.