Datasetpublic

BramVanroy/fineweb-2-duckdbs

DuckDB datasets for (dump, id) querying on FineWeb 2 This repo contains some DuckDB databases to check whether a given WARC UID exists in a FineWeb-2 dump. Usage example is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data), then you first have to extract the UID (the id column is of type UUID in the databases). Download All files: huggingface-cli download BramVanroy/fineweb-2-duckdbs --local-dir… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/fineweb-2-duckdbs.

sourceHugging Faceodc-byupdated 1y agoView on Hugging Face

0likes64kdownloads

Dataset Card

DuckDB datasets for (dump, id) querying on FineWeb 2

This repo contains some DuckDB databases to check whether a given WARC UID exists in a FineWeb-2 dump. Usage example is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data), then you first have to extract the UID (the id column is of type UUID in the databases).

Download

All files:

shell

huggingface-cli download BramVanroy/fineweb-2-duckdbs --local-dir duckdbs/fineweb-2/ --include *.duckdb --repo-type dataset

One file:

shell

huggingface-cli download BramVanroy/fineweb-2-duckdbs fw2-deu_Latn.duckdb --local-dir duckdbs/fineweb-2/ --repo-type dataset

Usage

Originally developed to be used with a library to look for Creative Commons licensing information: https://github.com/BramVanroy/CommonCrawl-CreativeCommons/

python

import re
import duckdb


uid_re = re.compile(r"<urn:uuid:([a-zA-Z0-9]{8}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{12})>")

duckdb_path = "duckdbs/fineweb-2/fw2-deu_Latn.duckdb"
con = duckdb.connect(duckdb_path, read_only=True)

dump = "CC-MAIN-2013-20"
uuid_urn = "<urn:uuid:4cd2db15-ae0c-482a-8688-d023d4b19f60>"
# !! Important: extract the UUID from the URN
uid = uid_re.sub("\\1", uuid_urn).replace("-", "")

query = "SELECT EXISTS (SELECT 1 FROM data WHERE dump = ? AND id = ?)"
exists = bool(con.execute(query, (dump, uid)).fetchone()[0])

print(f"Does ID {uid} exist in {dump}? {exists}")
# Does ID 4cd2db15ae0c482a8688d023d4b19f60 exist in CC-MAIN-2013-20? True