Datasetpublic

BramVanroy/fineweb-duckdbs

DuckDB datasets for id querying on FineWeb This repo contains DuckDB databases to check whether a given WARC UID exists in a FineWeb dump. Usage example is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data), then you first have to extract the UID (the id column is of type UUID in the databases). Download All files: huggingface-cli download BramVanroy/fineweb-duckdbs --local-dir duckdbs/fineweb/ --include… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/fineweb-duckdbs.

sourceHugging Faceodc-byupdated 1y agoView on Hugging Face

1likes2.8kdownloads

Dataset Card

DuckDB datasets for `id` querying on FineWeb

This repo contains DuckDB databases to check whether a given WARC UID exists in a FineWeb dump. Usage example is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data), then you first have to extract the UID (the id column is of type UUID in the databases).

Download

All files:

shell

huggingface-cli download BramVanroy/fineweb-duckdbs --local-dir duckdbs/fineweb/ --include *.duckdb --repo-type dataset

One file:

shell

huggingface-cli download BramVanroy/fineweb-duckdbs fw-CC-MAIN-2024-51.duckdb --local-dir duckdbs/fineweb/ --repo-type dataset

Usage

Originally developed to be used with a library to look for Creative Commons licensing information: https://github.com/BramVanroy/CommonCrawl-CreativeCommons/

python

import re
import duckdb


uid_re = re.compile(r"<urn:uuid:([a-zA-Z0-9]{8}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{12})>")

duckdb_path = "duckdbs/fineweb/fw-CC-MAIN-2019-30.duckdb"
con = duckdb.connect(duckdb_path, read_only=True)

uuid_urn = "<urn:uuid:93081b76-bc46-4d2f-98f0-2d9687b80967>"
# !! Important: extract the UUID from the URN
uid = uid_re.sub("\\1", uuid_urn).replace("-", "")

query = "SELECT EXISTS (SELECT 1 FROM data WHERE id = ?)"
exists = bool(con.execute(query, uid).fetchone()[0])

print(f"Does ID {uid} exist? {exists}")
# Does ID 93081b76bc464d2f98f0-2d9687b80967 exist? True