laurajul/SD_social

Step 1 - Scraping Data

• —Utilizes the CivitAI API with cursor-based pagination to fetch image data over a two-year period.
• —Saves progress in cursors.txt to resume scraping from the last retrieved point, avoiding redundant requests.
• —Stores data in timestamped directories, organizing results into manageable batches of 50,000 images per session.
• —Handles API constraints efficiently, with planned improvements for retrying failed requests.
• —

Step 2 - Normalizing Engagement Scores

Penalty (Time Penalty)

• —Purpose: To reduce the influence of older posts by applying a decay based on how long the content has been on the platform.
• —Formula: \( \text{timePenalty} = \frac{1}{1 + \log(\text{daysOnPlatform} + \text{offset})} \)
• —Logarithmic Decay: Older posts (higher daysOnPlatform) have smaller penalty values.
• —Offset: Ensures stability and avoids division by zero or undefined log values.
• —Effect:
• —Recent posts get higher scores.
• —Older posts are de-emphasized in engagement calculation.

Normalizing (Reactions Normalization)

• —Purpose: To scale raw social reaction values into a standardized range (0–1), enabling comparisons.
• —Method: Min-Max Scaling
• —Formula: \( \text{normalizedReactions} = \frac{\text{value} - \text{min}}{\text{max} - \text{min}} \)
• —Adjusts all reaction values relative to the minimum and maximum in the dataset.
• —Effect:
• —Converts raw reaction values into a uniform scale.
• —Ensures different ranges of reactions are treated fairly in further calculations.