Modelpublic

DATAGRID-research/DATAGRID-Open-Sora-Plan-v1.3.0-0.16M

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

1likes

Model Card

DATAGRID-Open-Sora-Plan-v1.3.0-0.16M

DATAGRID-Open-Sora-Plan-v1.3.0-0.16M is a Text-to-Video diffusion model based on the Open-Sora-Plan architecture. It has been fine-tuned by DATAGRID Inc. on a custom dataset of 0.16 million royalty-free video clips to generate high-quality videos from text prompts.

Model Details

—Developed by: DATAGRID Inc.
—Model type: Text-to-Video, Inpainting
—Languages: English
—License: Apache 2.0
—Finetuned from model: Open-Sora-Plan-v1.3.0

Model Description

This model extends the capabilities of Open-Sora-Plan by fine-tuning it on a curated, proprietary dataset.

Training Details

—Training Data: Fine-tuned on a custom dataset of 0.16 million royalty-free video-text pairs. This dataset was independently collected and curated by DATAGRID Inc., focusing on diverse scenes, motions, and objects. For V2V inpainting training data preparation, we built an automated mask generation pipeline utilizing state-of-the-art models like Meta AI's SAM2 (Segment Anything Model 2) and Microsoft's Florence2 to automatically generate masks for target objects in videos. This significantly improved efficiency and reduced costs compared to traditional manual annotation methods.

Inference Details

Our fork of Open-Sora-Plan with added mask handling capabilities for inpainting dataset and pipeline is available at DATAGRID-Research-org/Open-Sora-Plan. This fork extends the original model with improved inpainting functionality through enhanced mask processing.

Results

T2V

category	prompt	Open-Sora-Plan-v1.3.0	DATAGRID-Open-Sora-Plan-v1.3.0-0.16M
dynamic degree	an airplane accelerating to gain speed	[image]	[image]
object class	a bicycle	[image]	[image]
human action	A person is ice skating	[image]	[image]
color	A pink bird	[image]	[image]
imaging quality	this is how I do makeup in the morning	[image]	[image]
spatial relationship	a kite on the top of a skateboard, front view	[image]	[image]

V2V(Inpainting)

original video	prompt (short)	mask	DATAGRID-Open-Sora-Plan-v1.3.0-0.16M(Inpaint)
[image]	A juicer pours orange juice into a container. ・・・Morning juice-making scene.	[image]	[image]
[image]	A reddish-brown cat resting in a grassy field, ・・・relaxed and content.	[image]	[image]

License

This model is released under the Apache 2.0 License.

Citation

Citation information will be provided at a later date.