laurajul/SD_social

Step 1 - Scraping Data Utilizes the CivitAI API with cursor-based pagination to fetch image data over a two-year period. Saves progress in cursors.txt to resume scraping from the last retrieved point, avoiding redundant requests. Stores data in timestamped directories, organizing results into manageable batches of 50,000 images per session. Handles API constraints efficiently, with planned improvements for retrying failed requests. Step 2 - Normalizing Engagement… See the full description on the dataset page: https://huggingface.co/datasets/laurajul/SD_social.