datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
cc-re-2020-filtered
Auto-Generated FastDetector Dataset
Model Name: google/gemma-4-E4B-it
Sampling Params (as sent to the engine): {"temperature": 0.0, "top_p": 1.0, "presence_penalty": 0.0}
Ignored Params (unsupported by this engine): None
Prompt File: prompts/filter_contiguous_subset.json
Total Train Prompts: 1
Source Dataset: G-reen/cc-re-2020-raw-sharded
Source Column: text
Target Num Samples: all
Dropped Samples (over length limit 15000 tokens): 430
Failed API Requests: 495
Total… See the full description on the dataset page: https://huggingface.co/datasets/G-reen/cc-re-2020-filtered.ccrawl-recrawl-domains
Common Crawl Domain Recrawl
Live fetches of the home page of every ranked domain in Common Crawl's web graph, rendered to Markdown as they are fetched
What is it?
Common Crawl's web graph ranks domains by how central they are, but it does not tell you what those domains actually serve today. This dataset walks that ranking from the top and fetches each domain's home page now, storing the response as one Parquet row with the body, the headers, the timing and the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-domains.cc-re-2021-stat-val
Auto-Generated FastDetector Dataset
Dataset: G-reen/cc-re-2021-stat-val
Globals Config: config/globals_re2021.toml
Analysis Config: config/analysis_nofilter.toml
Rows: 9,822
Evaluation Results
Prompt Subsets: 4 (direct_reference, indirect_reference, revise, rewrite)
Generator Configs: 5 (Hy3-NVFP4-FP8 (Temp: 0.9), Llama-4-Scout-17B-16E-Instruct-NVFP4 (Temp: 0.6), Mixtral-8x7B-Instruct-v0.1 (Temp: 0.7), claude-sonnet-5 (Temp: Unknown), gpt-5.6-luna (Temp:… See the full description on the dataset page: https://huggingface.co/datasets/G-reen/cc-re-2021-stat-val.zoonomia-v1-v4_ccre_noexon
bolinas-dna/zoonomia-v1-v4_ccre_noexon
A curated enhancer training set for issue
#326 — a de-contaminated
derivation of the v4 ccre_non_promoter arm of
bolinas-dna/zoonomia-v1-v1,
built by the
snakemake/zoonomia_projection_dataset pipeline at commit
6b320c268547.
Provenance
This subset is the v4 ccre_non_promoter arm with every window that overlaps any other functional element (CDS / 3′UTR / ncRNA exon / TSS+5′UTR) removed — i.e. windows whose functional content… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v4_ccre_noexon.zoonomia-v1-v4_ccre_noexon_enhancer
bolinas-dna/zoonomia-v1-v4_ccre_noexon_enhancer
A curated enhancer training set for issue
#326 — a de-contaminated
derivation of the v4 ccre_non_promoter arm of
bolinas-dna/zoonomia-v1-v1,
built by the
snakemake/zoonomia_projection_dataset pipeline at commit
6b320c268547.
Provenance
This subset is v4_ccre_noexon further restricted to enhancer-dominant windows (dELS+pELS basepair coverage ≥ the other non-PLS cCRE classes), population-matching the val_enhancer… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v4_ccre_noexon_enhancer.zoonomia-v1-v3_ccre_non_promoter
bolinas-dna/zoonomia-v1-v3_ccre_non_promoter
Per-anchor region-type partition of the cross-mammal training set
bolinas-dna/zoonomia-v1-v1,
restricted to anchors labelled ccre_non_promoter by the
snakemake/zoonomia_projection_dataset pipeline
(commit 2ab868a2f1d4).
Region label (ccre_non_promoter)
ENCODE cCRE V4 non-promoter classes — cre_class != "PLS" (so: dELS, pELS, CA, CA-CTCF, CA-TF, CA-H3K4me3, TF), extended by 500 bp on each side. PLS is excluded because… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v3_ccre_non_promoter.ccrawl-urls
Common Crawl URL Index
Every URL Common Crawl has seen, as a slim columnar table, ready to seed a crawler frontier
What is it?
This dataset is the URL-level index of Common Crawl, republished as clean Parquet. Common Crawl is a non-profit that crawls the web every month and freely publishes its archives. Each crawl ships a columnar URL index that lists every captured page with its host, fetch status, content type, detected language, and a pointer into the WARC… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-urls.ccrawl-domains
Common Crawl Domain Ranks
Web domains ranked by harmonic centrality and PageRank, ready to prioritize a crawl
What is it?
This dataset is the domain-level ranking from Common Crawl's hyperlink web graph, republished as clean Parquet. Common Crawl builds a graph of which domains link to which, then scores every domain by harmonic centrality and PageRank. A high rank means many other well-connected domains link to it, which is a solid proxy for importance when you… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-domains.ccrawl-recrawl-urls
Common Crawl URL Recrawl
Live refetches of pages from Common Crawl's URL index, with the body inline and the text already extracted
What is it?
Common Crawl publishes which URLs it saw and when, but the page bodies live in WARC archives that are awkward to query and are as old as the crawl that made them. This dataset takes the URL index for a single monthly crawl and fetches the pages again now, storing each response as one Parquet row with the body, the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-urls.cc-re-2020-stat-train
Auto-Generated FastDetector Dataset
Dataset: G-reen/cc-re-2020-stat-train
Globals Config: config/globals_re2020.toml
Analysis Config: config/analysis_exclude_deepseek.toml
Rows: 108,377
Evaluation Results
Prompt Subsets: 4 (direct_reference, indirect_reference, revise, rewrite)
Generator Configs: 11 (Laguna-S-2.1-NVFP4 (Temp: 1.0), Llama-3.3-70B-Instruct-NVFP4 (Temp: 0.6), Llama-3.3-70B-Instruct-NVFP4 (Temp: 1.25), Mistral-Small-4-119B-2603-NVFP4 (Temp: 0.7)… See the full description on the dataset page: https://huggingface.co/datasets/G-reen/cc-re-2020-stat-train.ccrfcd-mrms-hrrr-env-2021-2025
1H gauge accumulation + MRMS/HRRR zarr dataset for the Desert Southwest
50+ MRMS+HRRR variables; 220+ gauges; 400k samples
NOTE: work in-progress.
This is a dataset for training and evaluating synthetic quantitative precipiation estimation (QPE) models. Given some input context (e.g., radar fields, envionrmental parameters), predict how much rain fell at a rain gauge site over some period of time. Concretely, we've gather and QC'd data from 220 tipping bucket gauges through… See the full description on the dataset page: https://huggingface.co/datasets/leharris3/ccrfcd-mrms-hrrr-env-2021-2025.zoonomia-v1-v4_ccre_non_promoter-order
bolinas-dna/zoonomia-v1-v4_ccre_non_promoter-order
The bolinas-dna/zoonomia-v1-v4_ccre_non_promoter cross-mammal
training set, restricted to a species cohort: one representative species per NCBI order — 19 deeply-diverged placental mammals (every pair separated by ~tens of millions of years), versus the implicit-default 108 family-deduplicated species. A strict subset of the family set, so it reuses the v1 cross-mammal projection unchanged (no re-halLiftover).
Same human anchors… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v4_ccre_non_promoter-order.cc-re-2021-filtered
Auto-Generated FastDetector Dataset
Model Name: google/gemma-4-E4B-it
Sampling Params (as sent to the engine): {"temperature": 0.0, "top_p": 1.0, "presence_penalty": 0.0}
Ignored Params (unsupported by this engine): None
Prompt File: prompts/filter_contiguous_subset.json
Total Train Prompts: 1
Source Dataset: G-reen/cc-re-2021-raw-sharded
Source Column: text
Target Num Samples: all
Dropped Samples (over length limit 15000 tokens): 70
Failed API Requests: 59
Total… See the full description on the dataset page: https://huggingface.co/datasets/G-reen/cc-re-2021-filtered.vertebrate-v1-issue473-fullwindow-ccre-enhancer-centered
marin-dna/vertebrate-v1-issue473-fullwindow-ccre-enhancer-centered
Review status: draft generated for issue #473 review before upload.
Human-anchored 255 bp vertebrate sequences for the
ccre_enhancer_centered cohort under the full_window policy.
The policy projects the complete 255 bp human window, applies the established 128--512 bp compatible-fragment gate, and resizes around the accepted target-span midpoint. Human anchors come from the fixed exp351 ENCODE dELS/pELS-centered… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/vertebrate-v1-issue473-fullwindow-ccre-enhancer-centered.zoonomia-v1-v4_ccre_noexon_enhancer-order
bolinas-dna/zoonomia-v1-v4_ccre_noexon_enhancer-order
The bolinas-dna/zoonomia-v1-v4_ccre_noexon_enhancer cross-mammal
training set, restricted to a species cohort: one representative species per NCBI order — 19 deeply-diverged placental mammals (every pair separated by ~tens of millions of years), versus the implicit-default 108 family-deduplicated species. A strict subset of the family set, so it reuses the v1 cross-mammal projection unchanged (no re-halLiftover).
Same human… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v4_ccre_noexon_enhancer-order.zoonomia-v1-v4_ccre_non_promoter
bolinas-dna/zoonomia-v1-v4_ccre_non_promoter
Per-anchor region-type partition of the cross-mammal training set
bolinas-dna/zoonomia-v1-v1,
restricted to anchors labelled ccre_non_promoter by the v4 region labeler
(snakemake/zoonomia_projection_dataset pipeline,
commit 4729d06d576f).
v4 re-derives the v3 partition with the labeling scheme resolved in
issue #221: base-pair
priority + window majority, a protein-coding-only TSS band, and
ccre_flank=0. See the pipeline README's "v4… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v4_ccre_non_promoter.vertebrate-v1-ccre_non_promoter
marin-dna/vertebrate-v1-ccre_non_promoter
Human-anchored 255 bp vertebrate sequences from the Zoonomia 447-mammal Cactus alignment and UCSC hg38 MultiZ 100-way alignment. This
draft covers the ccre_non_promoter region cohort with all species
scope and preserves source FASTA/2bit letter case.
Anchor eligibility uses the pipeline's pinned phyloP conservation filter.
Sequence case is independent of that filter: lowercase bases preserve source
repeat masking, uppercase bases… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/vertebrate-v1-ccre_non_promoter.cc-re-2021-temp-stat-train
WIP Fastdetector dataset
Waiting for statistics to finish generating...
vertebrate-v1-issue473-center1-ccre-enhancer-centered
marin-dna/vertebrate-v1-issue473-center1-ccre-enhancer-centered
Review status: draft generated for issue #473 review before upload.
Human-anchored 255 bp vertebrate sequences for the
ccre_enhancer_centered cohort under the center_1 policy.
The policy projects the exact central human nucleotide, requires one target locus, and emits the 255 bp target window centered on that mapped nucleotide. Human anchors come from the fixed exp351 ENCODE dELS/pELS-centered catalog after… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/vertebrate-v1-issue473-center1-ccre-enhancer-centered.cc-re-2020
cc-re-2020: pre-ChatGPT human-written English web text
4,998,498 documents of clean English, overwhelmingly human-written web text,
extracted from the 9 CommonCrawl
crawls of 2020, globally deduplicated within the year (and sharing no
identical document with the sister year), annotated with
WebOrganizer topic × format labels, and
selected so the per-format and per-topic row totals are each as uniform as the
eligible pool allows (water-filled marginal targets + exact LP; scarce… See the full description on the dataset page: https://huggingface.co/datasets/G-reen/cc-re-2020.cc-re-2020-raw-shardedcc-re-2020-filtered-labeledcc-re-2021
cc-re-2021: pre-ChatGPT human-written English web text
499,752 documents of clean English, overwhelmingly human-written web text,
extracted from the 9 CommonCrawl
crawls of 2021, globally deduplicated within the year (and sharing no
identical document with the sister year), annotated with
WebOrganizer topic × format labels, and
selected so the per-format and per-topic row totals are each as uniform as the
eligible pool allows (water-filled marginal targets + exact LP; scarce… See the full description on the dataset page: https://huggingface.co/datasets/G-reen/cc-re-2021.zoonomia-v1-v4_ccre_enhancer_centered-order
bolinas-dna/zoonomia-v1-v4_ccre_enhancer_centered-order
An enhancer-CENTERED training set for issue
#351, built by the
snakemake/zoonomia_projection_dataset pipeline
(workflow/rules/centered.smk) at commit
8127acfea5aa.
Provenance
Each training window is defined directly from an ENCODE cCRE V4 enhancer
(dELS + pELS): one 255 bp window centered on the cCRE midpoint
(make_enhancer_anchors, keep-all — clustered enhancers each keep their own
window), rather than a… See the full description on the dataset page: https://huggingface.co/datasets/marin-dna/zoonomia-v1-v4_ccre_enhancer_centered-order.coyo11m-256px-ccrop-latentcc-re-2021-raw-shardedccre-dart-eval
DART-Eval Task 1: cCRE Prioritization
This repository contains the hg38 parquet release of Task 1 from DART-Eval.
The task asks a model to distinguish ENCODE candidate cis-regulatory elements
(cCREs) from matched dinucleotide-shuffled controls.
Each source cCRE contributes two 350 bp sequences: the genomic sequence and a
control made by shuffling the bases within the cCRE while preserving
dinucleotide composition. The two rows share a pair_id, which supports the
paired zero-shot… See the full description on the dataset page: https://huggingface.co/datasets/Taykhoom/ccre-dart-eval.cCRE_gene_experiment_eQTLCCRDD150This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 150,
"total_frames": 106349,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:150"},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/LesserNeoguri/CCRDD150.eval_groot_ccrdd150_base_v1This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 9,
"total_frames": 10321,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:9"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/LesserNeoguri/eval_groot_ccrdd150_base_v1.