Josedde/image-to-video-human-preference-hailuo-02-marey

<style>

.vertical-container { display: flex; flex-direction: column; gap: 60px; }

.image-container img { height: 150px; / Set the desired height / margin:0; object-fit: contain; / Ensures the aspect ratio is maintained / width: auto; / Adjust width automatically based on height / }

.image-container { display: flex; / Aligns images side by side / justify-content: space-around; / Space them evenly / align-items: center; / Align them vertically / }

.container { width: 90%; margin: 0 auto; }

.text-center { text-align: center; display: flex; align-items:center; flex-direction: column; }

.score-amount { margin: 0; margin-top: 10px; }

.score-percentage { font-size: 12px; font-weight: semi-bold; }

</style>

Rapidata Video Generation Hailuo-02 v Marey Human Preference

<a href="https://www.rapidata.ai"> <img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization"> </a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback"> </a>

In this dataset, ~6k human responses from ~2k human annotators were collected to evaluate Seedance 1 Pro video generation model on our benchmark. This dataset was collected in roughtly 5 min using the Rapidata Python API, accessible to anyone and ideal for large scale data annotation.

Explore our latest model rankings on our website.

If you get value from this dataset and would like to see more in the future, please consider liking it ❤️

Overview

In this dataset, ~6k human responses from ~2k human annotators were collected to evaluate Seedance 1 Pro video generation model on our benchmark. This dataset was collected in roughtly 5 min using the Rapidata Python API, accessible to anyone and ideal for large scale data annotation.

Explanation of the colums

The dataset contains paired video comparisons. Each entry includes 'video1' and 'video2' fields, which contain links to downscaled GIFs for easy viewing. The full-resolution videos can be found here

The weighted_results column contains scores ranging from 0 to 1, representing aggregated user responses. Individual user responses can be found in the detailedResults column.

Alignment

The alignment score quantifies how well an video matches its prompt. Users were asked: "Which video fits the description better?".

Examples

<div class="vertical-container"> <div class="container"> <div class="text-center"> <q>A gentle breeze rustles the leaves and sways the grape cluster softly.</q> <img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/YQZcHv13mIghrsjwtZtY.jpeg" width=300> </div> <div class="image-container"> <div> <h3 class="score-amount">Marey </h3> <div class="score-percentage">(Score: 75.34%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey0059.gif" width=500> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Score: 24.66%)</div> <img src="https://assets.rapidata.ai/hailuo-02scene-motion0059.gif" width=500> </div> </div> </div> <div class="container"> <div class="text-center"> <q>The camera gently circles the blooming rose, capturing its petals in soft focus.</q> <img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/8YP7x40b3h4BlIqUaGxg-.jpeg" width=300> </div> <div class="image-container"> <div> <h3 class="score-amount">Marey </h3> <div class="score-percentage">(Score: 14.42%)</div> <img src="https://assets.rapidata.ai/marey0063.gif" width=500> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Score: 85.58%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02camera-motion_0063.gif" width=500> </div> </div> </div> </div>

Coherence

The coherence score measures whether the generated video is logically consistent and free from artifacts or visual glitches. Without seeing the original prompt, users were asked: "Which video has more glitches and is more likely to be AI generated?"

Examples

<div class="vertical-container"> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Marey </h3> <div class="score-percentage">(Glitch Rating: 13.68%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey0084.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Glitch Rating: 86.32%)</div> <img src="https://assets.rapidata.ai/hailuo-02scene-motion0084.gif" width="500" alt="Dataset visualization"> </div> </div> </div> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Marey </h3> <div class="score-percentage">(Glitch Rating: 88.92%)</div> <img src="https://assets.rapidata.ai/marey0003.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Glitch Rating: 11.08%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02style0003.gif" width="500" alt="Dataset visualization"> </div> </div> </div> </div>

Preference

The preference score reflects how visually appealing participants found each video, independent of the prompt. Users were asked: "Which video do you prefer aesthetically?"

Examples

<div class="vertical-container"> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Marey </h3> <div class="score-percentage">(Score: 78.21%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey0009.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Score: 21.79%)</div> <img src="https://assets.rapidata.ai/hailuo-02camera-motion0009.gif" width="500" alt="Dataset visualization"> </div> </div> </div> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Marey </h3> <div class="score-percentage">(Score: 11.02%)</div> <img src="https://assets.rapidata.ai/marey0020.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Score: 88.98%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02scene-motion0020.gif " width="500" alt="Dataset visualization"> </div> </div> </div> </div>

</br>

About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit rapidata.ai to learn more about how we're revolutionizing human feedback collection for AI development.

Other Datasets

We run a benchmark of the major video generation models, the results can be found on our website. We rank the models according to their coherence/plausiblity, their aligment with the given prompt and style prefernce. The underlying 2M+ annotations can be found here:

• —Link to the Rich Video Annotation dataset
• —Link to the Coherence dataset
• —Link to the Text-2-Image Alignment dataset
• —Link to the Preference dataset