trojblue/danbooru2025-metadata

<p align="center"> <img src="https://huggingface.co/datasets/trojblue/danbooru2025-metadata/resolve/main/57931572.png" alt="Danbooru Logo" width="120"/> </p>

<h1 align="center">🎨 Danbooru 2025 Metadata</h1>

<p align="center"> <strong>Latest Post ID:</strong> <code>9,158,800</code><br/> <em>(as of Apr 16, 2025)</em> </p>

📁 About the Dataset This dataset provides structured metadata for user-submitted images on Danbooru, a large-scale imageboard focused on anime-style artwork.

Scraping began on January 2, 2025, and the data are stored in Parquet format for efficient programmatic access. Compared to earlier versions, this snapshot includes:

• —More consistent tag history tracking
• —Better coverage of older or previously skipped posts
• —Reduced presence of unlabeled AI-generated entries

Dataset Overview

Each row corresponds to a Danbooru post, with fields including:

• —Tag list (both general and system-specific)
• —Upload timestamp
• —File details (size, extension, resolution)
• —User stats (favorites, score, etc.)

The schema follows Danbooru’s public API structure, and should be familiar to anyone who has worked with their JSON output.

File Format

The metadata are stored in a flat table. Nested dictionaries have been flattened using a consistent naming scheme (parentkey_childkey) to aid downstream use in ML pipelines or indexing tools.

Access & Usage

You can load the dataset via the Hugging Face datasets library:

from datasets import load_dataset
danbooru_metadata = load_dataset("trojblue/danbooru2025-metadata", split="train")
df = danbooru_metadata.to_pandas()

Potential use cases include:

• —Image retrieval systems
• —Text-to-image alignment tasks
• —Dataset curation or filtering
• —Historical or cultural analysis of trends in tagging

Be cautious if working in public settings. The dataset contains adult content.

Notable Characteristics

• —Single-Snapshot Coverage: All posts up to the stated ID are included. No need to merge partial scrapes.
• —Reduced Tag Drift: Many historic tag renames and merges are reflected correctly.
• —Filtered AI-Generated Posts: Some attempts were made to identify and exclude unlabeled AI-generated entries, though the process is imperfect.

Restricted tags (e.g., certain content filters) are inaccessible without privileged API keys and are therefore missing here.

If you need metadata with those tags, you’ll need to integrate previous datasets (such as Danbooru2021) and resolve inconsistencies manually.

Code: Flattening the JSON

Included below is a simplified example showing how the raw JSON was transformed:

import pandas as pd
from pandarallel import pandarallel

# Initialize multiprocessing
pandarallel.initialize(nb_workers=4, progress_bar=True)

def flatten_dict(d, parent_key='', sep='_'):
    items = []
    for k, v in d.items():
        new_key = f"{parent_key}{sep}{k}" if parent_key else k
        if isinstance(v, dict):
            items.extend(flatten_dict(v, new_key, sep=sep).items())
        elif isinstance(v, list):
            items.append((new_key, ', '.join(map(str, v))))
        else:
            items.append((new_key, v))
    return dict(items)

def extract_all_illust_info(json_content):
    return pd.Series(flatten_dict(json_content))

def dicts_to_dataframe_parallel(dicts):
    df = pd.DataFrame(dicts)
    return df.parallel_apply(lambda row: 

extract_all_illust_info(row.to_dict()), axis=1)

Warnings & Considerations

• —NSFW Material: Includes sexually explicit tags or content. Do not deploy without clear filtering and compliance checks.
• —Community Bias: Tags are user-generated and reflect collective subjectivity. Representation may skew or omit.
• —Data Licensing: Image rights remain with original uploaders. This dataset includes metadata only, not media. Review Danbooru’s Terms of Service for reuse constraints.
• —Missing Content: Posts with restricted tags or deleted content may appear with incomplete fields or be absent entirely.

Column Summaries (Sample — Apr 16, 2025)

Full schema and additional statistics are viewable on the Hugging Face Dataset Viewer.

File Information

• —file_url: 8.8 million unique file links
• —file_ext: 9 file types
• —'jpg': 73.3%
• —'png': 25.4%
• —Other types (mp4, gif, zip, etc.): <1.5% combined
• —file_size and media_asset_file_size (bytes):
• —Min: 49
• —Max: ~106MB
• —Avg: ~1.5MB

Image Dimensions

• —image_width:
• —Min: 1 px
• —Max: 35,102 px
• —Mean: 1,471 px
• —image_height:
• —Min: 1 px
• —Max: 54,250 px
• —Mean: 1,760 px

(Note: extremely small dimensions may indicate deleted or broken images.)

Scoring and Engagement

• —score (net = up − down):
• —Min: −167
• —Max: 2,693
• —Mean: 26.15
• —up_score:
• —Max: 2,700
• —Mean: 25.87
• —down_score:
• —Min: −179
• —Mean: −0.24
• —fav_count:
• —Max: 4,458
• —Mean: 32.49

Rating and Moderation

• —rating:
• —'g' (general, safe): 29.4%
• —'s' (suggestive): 49.5%
• —'q' (questionable): 11.2%
• —'e' (explicit): 9.8%
• —is_banned: 1.13% true
• —is_deleted: 5.34% true
• —is_flagged / is_pending: <0.01% true (rare moderation edge-cases)

Children & Variations

• —has_children: 10.7%
• —has_active_children: 10.0%
• —has_visible_children: 10.3%
• —has_large: 70.6% of posts are linked to full-res versions

Tag Breakdown

(Tag counts are per post; some posts may have hundreds.)

• —tag_count (total tags):
• —Avg: 36.3
• —tag_count_artist: 0.99 avg
• —tag_count_character: 1.62 avg
• —tag_count_copyright: 1.39 avg
• —tag_count_general: 30.0 avg
• —tag_count_meta: 2.3 avg

Some outliers contain hundreds of tags—up to 1,250 in total on rare posts.

Other Fields

• —uploader_id (anonymized integer ID)
• —updated_at (timestamp) — nearly every post has a unique update time

(last updated: 2025-04-16 12:15:29.262308)