opencapybara/CapyWiki-34M-raw

CapyWiki-34M

CapyWiki is a collection of openly licensed and public domain image datasets from Wikimedia. Raw means unfiltered: no aesthetic or filtering, so many book scans and other images are in place.

CapyWiki contains 3 splits:

• —public_domain split: 16.5M links to Wikimedia images that were categorized with license info of Public Domain, cc0 or equivalent. There are no restrictions on how this images can be used from a copyright standpoint.
• —cc_by split: 3.4M links to Wikimedia images that have a commercial usage permissive cc-by or equivalent license.
• —cc_by_sa split: 14.2M links to Wikimedia images that have a commercial usage permissive cc-by-sa or equivalent license.

The dataset should contain photos, illustrations, scans, maps and any other media categories in an image format that Wikimedia hosts.

Loading the dataset for downstream use-cases

The dataset splis are in a *.parquet format and can be read/processed by any tool or library that can read *.parquet files.

If you wish to use the Hugging Face Datasets library you can load the dataset as:

from datasets import load_dataset
#Load the public_domain split
dataset = load_dataset("opencapybara/CapyWiki-34M", split="public_domain")

#Now the dataset can be used for any downstream cases e.g.:
# first_500_urls = dataset[:500]['url']

You can use the img2dataset to download the images from this dataset.

What's the intended use of CapyWiki

Using CapyWiki to train and evaluate neural networks is possible but not exclusive.

The public domain image split can be used without freely, openly and without any restrictions from a copyright standpoint, while the cc-by and cc-by-sa have the provisions indicated on each license.

The information contained on this Model Card is not legal advice and we recommend you conducting your own independent analysis of the content and its copyright status.

Data format dictionary

The dataset contains:

• —url for the image URL
• —description as the original image description (may include original wikimedia HTML tags)
• —author as an HTML link tag for whoever is indicated as an author in wikimedia
• —license spelled out license name text
• —license_wiki shortened license nickname
• —date artefact date
• —credit credit

Help needed

This is a raw dataset. Tasks such as: captioning, content classification (photos, illustrations, etc.), aesthetic classifications, meta-data inclusion (width, height) or the images are open and community contributions for those are more than welcome.