Datasetpublic

6DammK9/danbooru2023-captions-1ktar

Danbooru 2023 captions only in 1k tar Raw captions jointed by unpublished extended dataset from KBlueLeaf/danbooru2023-metadata-database It aligns to nyanko7/danbooru2023. There are around 200k missing for the 2024 version, I'll try to use Minthy/ToriiGate-v0.4-7B to fill in the rest. meta_cap.json has been provided in compressed format if you want to train with kohyas triner. Currently I'm trying to merge this with my 2024 version. Core logic The script… See the full description on the dataset page: https://huggingface.co/datasets/6DammK9/danbooru2023-captions-1ktar.

sourceHugging Facemitupdated 2y agoView on Hugging Face

0likes2downloads

Dataset Card

Danbooru 2023 captions only in 1k tar #

—Raw captions jointed by unpublished extended dataset from KBlueLeaf/danbooru2023-metadata-database

—It aligns to nyanko7/danbooru2023. There are around 200k missing for the 2024 version, I'll try to use Minthy/ToriiGate-v0.4-7B to fill in the rest.

—meta_cap.json has been provided in compressed format if you want to train with kohyas triner. Currently I'm trying to merge this with my 2024 version.

Core logic ##

—The script building this 1ktar

—Artist and character first. Others can rely on pretrained knowledge. No fancy "quality" score for [pretraining](https://techcommunity.microsoft.com/blog/machinelearningblog/differences-between-pre-training-and-supervised-fine-tuning-sft/4220673) focused dataset.

rearranged_tags = [row.phi3v_horny, row.pixtral_caption, row.florence_short, row.florence_long]
must_exist = [tag for tag in rearranged_tags if tag]
tag = " ".join(must_exist)

How to build the "dataset" with speed ##

—Refer to the tags-1ktar repo.

—This repo is still in develeopment. My current task of building the dataset for "pretraining".