Modelpublic

Echo-Team/Echo-WM-Base-Diffusers

sourceHugging Faceotherupdated 18d agoView on Hugging Face

0likes69downloads

Model Card

Echo-WM Base — Diffusers Format

This repository provides the Diffusers-format version of Echo-WM Base. The weights are converted from the official `Echo-Team/Echo-WM` checkpoint, with the model architecture and inference workflow adapted to the Diffusers Modular Pipeline API.

Echo-WM jointly generates video and synchronized environmental sound, music, and speech from an initial image, a text prompt, and a camera-action sequence.

Diffusers components

—EchoWMTransformer3DModel
—EchoWMModularPipeline
—EchoWMBlocks

Installation

bash

pip install -U diffusers transformers accelerate av

Usage

The following example uses the official `wm_cases/0010` input and camera trajectory.

python

import torch
from diffusers import ComponentsManager, ModularPipeline
from diffusers.utils import encode_video, load_image

components_manager = ComponentsManager()
components_manager.enable_auto_cpu_offload(device="cuda")
pipe = ModularPipeline.from_pretrained(
    "Echo-Team/Echo-WM-Base-Diffusers", components_manager=components_manager
)
pipe.load_components(dtype=torch.bfloat16)

image = load_image(
    "https://raw.githubusercontent.com/jd-opensource/JoyAI-Echo/main/echo_wm/examples/wm_cases/0010/input.png"
)
prompt = """Environment: A fantasy canyon with a teal pool, pale cliffs, and a distant palace.
Character: A solitary adventurer in a teal cloak, seen from behind.
Style: Painterly cinematic fantasy with soft golden daylight.
Perspective: Wide third-person rear view at standing height.
Sounds: Water laps against the shore, birds echo between the cliffs, and leaves move in a light breeze.
Speech: None."""

result = pipe(
    image=image,
    prompt=prompt,
    negative_prompt="blurry, jittery, text, watermark",
    action="w-60,a-60,w-60,d-60",
    num_frames=241,
    num_inference_steps=30,
    generator=torch.Generator(device="cuda").manual_seed(34),
    output=["videos", "audio"],
)

encode_video(
    video=result["videos"][0],
    audio=result["audio"][0].float().cpu(),
    fps=24,
    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
    output_path="echo_wm_av.mp4",
)

Optional action overlay

The pipeline returns clean frames. To draw a WASD/IJKL HUD, apply the Echo-WM postprocessing utility before encoding:

python

from diffusers.modular_pipelines.echo_wm import apply_action_overlay

video = apply_action_overlay(result["videos"][0], action="w-60,a-60,w-60,d-60")
encode_video(
    video=video,
    audio=result["audio"][0].float().cpu(),
    fps=24,
    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
    output_path="echo_wm_action_av.mp4",
)

Camera controls

Each action segment uses the form <keys>-<duration>. W/S move forward or backward, A/D strafe, I/K pitch, J/L yaw, and none holds the camera still. Segments are joined with commas, for example w-60,a-60,w-60,d-60.

Original project

Citation

bibtex

@article{zhang2026echowm,
  title         = {EchoWM: Open and Enterable Omnimodal World Models},
  author        = {Zhang, Songchun and Li, Yaowei and Zhuang, Junhao and Jin, Weiyang and Wang, Haoyu and Lu, Xin and Sun, Yilang and Zhang, Shiyi and Li, Haoran and Ma, Xiaoxiao and Li, Yuming and Liu, Yijun and Su, Yaofeng and Ma, Yanwen and Wu, Haoyu and Su, Zihan and Ma, Yue and Zhang, Lvmin and Huang, Haoyang and Xue, Zeyue and Rao, Anyi and Duan, Nan},
  journal       = {arXiv preprint arXiv:2608.23189},
  year          = {2026},
  eprint        = {2608.23189},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}

License

Academic research and non-commercial use only. This model follows the LTX-2 Community License.