argilla/ultrafeedback-binarized-curation

Ultrafeedback binarized dataset using the mean of preference ratings

Introduction

This dataset contains the result of curation work performed by Argilla (using Argilla 😃).

After visually browsing around 200 examples using the sort and filter feature of Argilla, we noticed a strong mismatch between the overall_score in the original UF dataset (and the Zephyr train_prefs dataset) and the quality of the chosen response.

By adding the critique rationale to our Argilla Dataset, we confirmed the critique rationale was highly negative, whereas the rating was very high (the highest in fact: 10). See screenshot below for one example of this issue. After some quick investigation, we identified hundreds of examples having the same issue and a potential bug on the UltraFeedback repo.

For context, this is the corresponding example within the train_prefs dataset with a score_chosen of 10.

The dataset is fully open and browsable at https://huggingface.co/spaces/argilla/ultrafeedback-curator (credentials: owner/12345678). Try browsing by discarded or using the sort feature to find problematic records yourself.

Dataset processing

1. 1.We have identified a buggy behaviour of how overall_score was generated in the UltraFeedback dataset using the Critique Model, which caused very low quality (and rated) responses to get a very high score. The reason is this line which will give a `10` to responses that get a `1` from the Critique model.
2. 2.To benefit from the preference data of UltraFeedback (aspect-based preference data: honesty, instruction-following, etc.) and not the Critique model (which evaluates a single response individually), we have opted for not using overall_score and compute the mean of preference ratings instead.
3. 3.We select the best reponse based on this mean (named best_rated_response), and keep the one based on the overall_score for comparison purposes
4. 4.We select a random response with lower mean rating (or equal in the worst case scenario, for preference tuning we'll filter those cases out), named random_response_for_best_rated. This follows the method described in the Zephyr paper of picking a random response instead of the lowest rated response. In any case, we keep all completions for people looking at additional approaches. One could binarize the data differently, for example generating several pairs per row based on their ranking (as done on the OpenAI work).
5. 5.We have tried to keep all additional data for reproducibility.

Please note that *_best_overall scores are in the [1,10] range and *_best_rated are in the [1,5] range.

Based on an initial analysis, using mean rating vs overallscore picks a different chosen response in ~30K examples (out of ~63K). Additionally, using overallscore results in picking responses from less powerful models more often. See the distribution below: