Rapidata/text-2-video-human-preferences-sora-2

<style>

.vertical-container { display: flex; flex-direction: column; gap: 60px; }

.image-container img { height: 150px; / Set the desired height / margin:0; object-fit: contain; / Ensures the aspect ratio is maintained / width: auto; / Adjust width automatically based on height / }

.image-container { display: flex; / Aligns images side by side / justify-content: space-around; / Space them evenly / align-items: center; / Align them vertically / }

.container { width: 90%; margin: 0 auto; }

.text-center { text-align: center; }

.score-amount { margin: 0; margin-top: 10px; }

.score-percentage { font-size: 12px; font-weight: semi-bold; }

</style>

Rapidata Video Generation Sora 2 Human Preference

<a href="https://www.rapidata.ai"> <img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization"> </a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback"> </a>

In this dataset, ~75k human responses from ~15k human annotators were collected to evaluate the Sora 2 video generation model on our benchmark. This dataset was collected in roughtly 30 min using the Rapidata Python API, accessible to anyone and ideal for large scale data annotation.

Explore our latest model rankings on our website.

If you get value from this dataset and would like to see more in the future, please consider liking it ❤️

Overview

In this dataset, ~75k human responses from ~15k human annotators were collected to evaluate the Sora 2 video generation model on our benchmark. This dataset was collected in roughtly 30 min using the Rapidata Python API, accessible to anyone and ideal for large scale data annotation. The benchmark data is accessible on huggingface directly.

Explanation of the colums

The dataset contains paired video comparisons. Each entry includes 'video1' and 'video2' fields, which contain links to downscaled GIFs for easy viewing. The full-resolution videos can be found here

The weighted_results column contains scores ranging from 0 to 1, representing aggregated user responses. Individual user responses can be found in the detailedResults column.

Alignment

The alignment score quantifies how well an video matches its prompt. Users were asked: "Which video fits the description better?".

Examples

<div class="vertical-container"> <div class="container"> <div class="text-center"> <q>A split-screen showing different natural phenomena occurring simultaneously. Capture lightning striking mountains, waves crashing on cliffs, forests swaying in the wind, and clouds racing across the sky. Highlight the raw power, beauty, and harmony of nature's elements.</q> </div> <div class="image-container"> <div> <h3 class="score-amount">Sora2 </h3> <div class="score-percentage">(Score: 100%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora29-10-25870.gif" width=500> </div> <div> <h3 class="score-amount">Alpha </h3> <div class="score-percentage">(Score: 0%)</div> <img src="https://assets.rapidata.ai/alpha00871286196315.gif" width=500> </div> </div> </div> <div class="container"> <div class="text-center"> <q>A serene dawn view of a lone wolf traversing a misty forest, its fur illuminated by the early light as it moves gracefully through towering trees, embodying both solitude and strength.</q> </div> <div class="image-container"> <div> <h3 class="score-amount">Sora2 </h3> <div class="score-percentage">(Score: 0%)</div> <img src="https://assets.rapidata.ai/sora29-10-25540.gif" width=500> </div> <div> <h3 class="score-amount">Sora </h3> <div class="score-percentage">(Score: 100%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora00540.gif" width=500> </div> </div> </div> </div>

Coherence

The coherence score measures whether the generated video is logically consistent and free from artifacts or visual glitches. Without seeing the original prompt, users were asked: "Which video has more glitches and is more likely to be AI generated?"

Examples

<div class="vertical-container"> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Sora2 </h3> <div class="score-percentage">(Glitch Rating: 0%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora29-10-2550.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Hunyuan </h3> <div class="score-percentage">(Glitch Rating: 100%)</div> <img src="https://assets.rapidata.ai/hunyuan0005421.gif" width="500" alt="Dataset visualization"> </div> </div> </div> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Sora2 </h3> <div class="score-percentage">(Glitch Rating: 90.61%)</div> <img src="https://assets.rapidata.ai/sora29-10-25750.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Veo2 </h3> <div class="score-percentage">(Glitch Rating: 9.39%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo200750.gif" width="500" alt="Dataset visualization"> </div> </div> </div> </div>

Preference

The preference score reflects how visually appealing participants found each video, independent of the prompt. Users were asked: "Which video do you prefer aesthetically?"

Examples

<div class="vertical-container"> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Sora2 </h3> <div class="score-percentage">(Score: 95.44%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora29-10-2540.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Mochi 1 </h3> <div class="score-percentage">(Score: 4.56%)</div> <img src="https://assets.rapidata.ai/mochi-1-24-7-2540.gif" width="500" alt="Dataset visualization"> </div> </div> </div> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Sora2 </h3> <div class="score-percentage">(Score: 10.16%)</div> <img src="https://assets.rapidata.ai/sora29-10-25640.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">seedance 1 pro </h3> <div class="score-percentage">(Score: 89.84%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/seedance-1-pro-24-7-25640.gif" width="500" alt="Dataset visualization"> </div> </div> </div> </div>

</br>

About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit rapidata.ai to learn more about how we're revolutionizing human feedback collection for AI development.

Other Datasets

We run a benchmark of the major video generation models, the results can be found on our website. We rank the models according to their coherence/plausiblity, their aligment with the given prompt and style prefernce. The underlying 2M+ annotations can be found here:

• —Link to the Rich Video Annotation dataset
• —Link to the Coherence dataset
• —Link to the Text-2-Image Alignment dataset
• —Link to the Preference dataset