Datasetpublic

webshart/e621-2024-webp-4Mpixel-webshart-indices

e621-2024-webp-4Mpixel webshart indices (shards 0-48) Per-shard webshart index files for NebulaeWis/e621-2024-webp-4Mpixel, covering tar shards original/data-0000.tar through original/data-0048.tar (193,966 samples). Each entry carries the tar member byte offset/length, image geometry (width, height, aspect), and captions merged from RareConcepts/e621-2024-metadata-4M-recap-qwen2.5-vl-7B (qwen2.5-VL-7B recaptions). Coverage is deliberately partial (first 49 of 1000 shards, built… See the full description on the dataset page: https://huggingface.co/datasets/webshart/e621-2024-webp-4Mpixel-webshart-indices.

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes49downloads

Dataset Card

e621-2024-webp-4Mpixel webshart indices (shards 0-48)

Per-shard webshart index files for NebulaeWis/e621-2024-webp-4Mpixel, covering tar shards original/data-0000.tar through original/data-0048.tar (193,966 samples). Each entry carries the tar member byte offset/length, image geometry (width, height, aspect), and captions merged from RareConcepts/e621-2024-metadata-4M-recap-qwen2.5-vl-7B (qwen2.5-VL-7B recaptions).

Coverage is deliberately partial (first 49 of 1000 shards, built for a ~131k-sample training subset). Extend with:

bash

webshart extract-metadata \
  --source NebulaeWis/e621-2024-webp-4Mpixel \
  --destination <dir> \
  --range 49,1000 \
  --include-image-geometry

then merge captions with webshart.write_captions_to_metadata using the shard + filename columns of the RareConcepts metadata parquet files.

SimpleTuner usage

json

{
    "id": "e621-webshart",
    "type": "webshart",
    "dataset_type": "image",
    "source": "NebulaeWis/e621-2024-webp-4Mpixel",
    "subfolder": "original",
    "metadata": "RareConcepts/e621-2024-webp-4Mpixel-webshart-indices",
    "caption_strategy": "webshart",
    "metadata_backend": "webshart"
}