datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Torrent_Copyright_Mappingsfinewebs-copyright-domains
List of domains that were removed from FineWeb(-2)
An easy-access dataset that contains the domains that were removed from FineWeb(-2) after take-down requests. This should allow others to quickly filter out domains that should not be included in data collection, removing redundant data processing. If used in this way, as a domain-wide filter, it should make life easier for the copyright holders, too, who will not have to resubmit cease-and-desists for each new webcrawl.
Currently… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/finewebs-copyright-domains.bhl_copyright_statuses
Biodiversity Heritage Library Copyright Statuses
This dataset contains all unique copyright statuses present in the items.txt.gz file of the Biodiversity Heritage Library open dataset on AWS Open Data. The unique copyright statuses were extracted, grouped and sorted by frequency using the following DuckDB query:
COPY (SELECT CopyrightStatus, COUNT(*) as Count FROM read_csv('https://bhl-open-data.s3.amazonaws.com/data/item.txt.gz') GROUP BY CopyrightStatus ORDER BY Count DESC) TO… See the full description on the dataset page: https://huggingface.co/datasets/storytracer/bhl_copyright_statuses.