Datasetpublic

6DammK9/danbooru2024-captions-1ktar

Danbooru 2024 captions only in 1k tar Raw captions jointed by 7.62M unpublished extended dataset from KBlueLeaf/danbooru2023-metadata-database and 0.48M generated dataset via Minthy/ToriiGate-v0.4-7B in exl2-8bpw mode. There are 8.13M in total. python convert_meta_to_tar.py Reading source JSON Keys count: 8136011 max id: 8360499 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000… See the full description on the dataset page: https://huggingface.co/datasets/6DammK9/danbooru2024-captions-1ktar.

sourceHugging Facemitupdated 2y agoView on Hugging Face

3likes179downloads

Dataset Card

Danbooru 2024 captions only in 1k tar #

—Raw captions jointed by 7.62M unpublished extended dataset from KBlueLeaf/danbooru2023-metadata-database and 0.48M generated dataset via Minthy/ToriiGate-v0.4-7B in exl2-8bpw mode. There are 8.13M in total.

log

python convert_meta_to_tar.py
Reading source JSON
Keys count: 8136011
max id: 8360499
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:32<00:00,  2.21it/s]
Files written: 1000

—It aligns to deepghs/danbooru2024.

log

>python merge_tag_and_caption_to_meta.py
loading JSON_ID
loading JSON_TAGS
loading JSON_CAPTION
loading JSON_LATENT
start merging
merging json files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 8005010/8005010 [00:51<00:00, 155076.84it/s]
ids: 8005010, missing tags: 0, missing caption: 0, missing latent: 0
writing OUTPUT_JSON
writing MISSING_JSON
Merge complete.

—meta_cap.json has been provided in compressed format if you want to train with kohyas triner. I have a "tag with caption" which is 14GB decompressed..

Core logic ##

—The scripts / logs / docuements building this 1ktar, majroly batch_nlp_caption_exl.py

—Artist and character first. Others can rely on pretrained knowledge. No fancy "quality" score for [pretraining](https://techcommunity.microsoft.com/blog/machinelearningblog/differences-between-pre-training-and-supervised-fine-tuning-sft/4220673) focused dataset.

rearranged_tags = [row.phi3v_horny, row.pixtral_caption, row.florence_short, row.florence_long]
must_exist = [tag for tag in rearranged_tags if tag]
tag = " ".join(must_exist)

—It is also reflected in nlp_stuffs.py.

How to build the "dataset" with speed ##

—Refer to the tags-1ktar repo.

—This repo is still in develeopment. My current task of building the dataset for "pretraining".

—Hint: The files are in *.caption instead of *.txt. It should refers to default settings.