Datasetpublic

6DammK9/e621_2024-captions-1ktar

E621 2024 captions only in 1k tar Raw captions jointed from lodestones/e621-captions It doesn't align to any dataset yet. meta_cap.json has been provided in compressed format if you want to train with kohyas triner. Currently I'm trying to merge this with my 2024 version. Core logic The script building this 1ktar There is not much choice, I don't have GPU to run for 1M captions with VLM so I just "take it or leave it". rearranged_tags =… See the full description on the dataset page: https://huggingface.co/datasets/6DammK9/e621_2024-captions-1ktar.

sourceHugging Facemitupdated 2y agoView on Hugging Face

0likes17downloads

Dataset Card

E621 2024 captions only in 1k tar #

—Raw captions jointed from lodestones/e621-captions

—It doesn't align to any dataset yet.

—meta_cap.json has been provided in compressed format if you want to train with kohyas triner. Currently I'm trying to merge this with my 2024 version.

Core logic ##

—The script building this 1ktar

—There is not much choice, I don't have GPU to run for 1M captions with VLM so I just "take it or leave it".

rearranged_tags = [row.regular_summary, row.brief_summary] if row.response_finish_reason == "STOP" else [row.tags]
must_exist = [tag for tag in rearranged_tags if tag]
caption = " ".join(must_exist).replace("\n","").replace("\r","")

How to build the "dataset" with speed ##

—Refer to the tags-1ktar repo.

—This repo is still in develeopment. My current task of building the dataset for "pretraining".