CoolFace
Datasetpublic

scarysnake/outfitter-advice

outfitter-advice On the OUTFITS subreddit, users frequently post multiple images and ask commenters to pick one (with a numeric comment like "2") This evaluation collects the top two comments and their relative scores (for example, if there is overwhelming support for one option over another, your evaluation should score differently from a close match). The first and second images in the "images" field are the best and second-best images. We shuffle them before evaluating. If… See the full description on the dataset page: https://huggingface.co/datasets/scarysnake/outfitter-advice.

sourceHugging Facemitupdated 8mo agoView on Hugging Face
1likes29downloads
Dataset Card

outfitter-advice

On the OUTFITS subreddit, users frequently post multiple images and ask commenters to pick one (with a numeric comment like "2")

This evaluation collects the top two comments and their relative scores (for example, if there is overwhelming support for one option over another, your evaluation should score differently from a close match).

The first and second images in the "images" field are the best and second-best images. We shuffle them before evaluating. <br/> If there is overwhelming support for the top choice, the second-best will be arbitrary and given 1 vote.

Standards and Practices

Rules of the Road

  • —This should only evaluate posts with Male, Female, or Nonbinary Advice flair (no unsolicited advice)
  • —Remove posts from the dataset if the image, user, or post is deleted
  • —Do not post or comment on the subreddit
  • —Do not upload the images separately from Reddit

Technical Details

  • —Only pick posts which select between multiple outfits (not freeform advice, shoes, etc)
  • —Limited string replace support for first, second, third, last, etc.
  • —A top comment which mentions multiple outfits is not parseable, so the post is not parseable
  • —This assumes that a comment supports the outfit which they mentioned
  • —This assumes that the user posted one image per outfit (i.e. outfit 3 = third image)
  • —If there is overwhelming support for the top choice, the second choice will be arbitrary and given 1 vote.

Code

Python script preparing /r/OUTFITS posts for dataset: scraper.py

Python script testing OpenAI's o4-mini model: eval_openai.py

Evaluation Scores

Currently scoring by:

  • —1 point for best choice
  • —secondChoice/firstChoiceVotes for second-best choice
  • —0 points for other options

OpenAI's o4-mini: 8.875 / 12

With more options we should be able to break down scores by M/F/NB flair.

Dependencies

For adding to evaluation set:<br/> Requires a Reddit appid, secret, and useragent saved in .env in JSON format. See https://praw.readthedocs.io/en/stable/gettingstarted/quickstart.html

  • —praw

For evaluating with OpenAI:<br/>

  • —openai