CoolFace
Modelpublic

Echo-Team/tiny-echo-wm-base-diffusers

sourceHugging Faceupdated 18d agoView on Hugging Face
0likes38downloads
Model Card

This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

Pipeline Type: EchoWMBlocks

Description:

This pipeline uses a 5-block architecture that can be customized and extended.

Example Usage

[TODO]

Pipeline Architecture

This modular pipeline is composed of the following blocks:

  1. 1.text (LTX2TextConditioningStep)
  2. 2.Text-conditioning stage for LTX-2.X: encodes the prompt(s), then runs the text connectors to produce the video/audio-branch connector embeddings the denoiser consumes. Outputs stay at one row per prompt -- the denoise stage expands them by num_videos_per_prompt -- so they can be reused across denoise runs.
  3. 3.camera (EchoWMCameraConditionStep)
  4. 4.image_encoder (EchoWMVaeEncoderStep)
  5. 5.VAE encoder step that encodes the input image into normalized latents for image-to-video generation.
  6. 6.denoise (EchoWMImage2VideoCoreDenoiseStep)
  7. 7.Denoise block (image-to-video) that expands the text conditioning by num_videos_per_prompt, adds image conditioning and runs the joint denoising loop.
  8. 8.decode (EchoWMDecoderStep)

Model Components

  1. 1.text_encoder (PreTrainedModel)
  2. 2.tokenizer (PreTrainedTokenizerBase)
  3. 3.connectors (LTX2TextConnectors)
  4. 4.transformer (EchoWMTransformer3DModel)
  5. 5.vae (AutoencoderKLLTX2Video)
  6. 6.video_processor (VideoProcessor)
  7. 7.scheduler (FlowMatchEulerDiscreteScheduler)
  8. 8.audio_vae (AutoencoderKLLTX2Audio)
  9. 9.guider (LTX2Guidance)
  10. 10.audio_guider (LTX2Guidance)
  11. 11.vocoder (LTX2Vocoder)

Input/Output Specification

Inputs:

  • —prompt (str): The prompt or prompts to guide image generation.
  • —negative_prompt (str, optional): The prompt or prompts not to guide the image generation.
  • —max_sequence_length (int, optional, defaults to 1024): Maximum sequence length for prompt encoding.
  • —action (str): WASD/IJKL action program.
  • —height (int, optional, defaults to 704): The height in pixels of the generated image.
  • —width (int, optional, defaults to 1280): The width in pixels of the generated image.
  • —num_frames (int, optional, defaults to 241): Number of output video frames.
  • —frame_rate (float, optional, defaults to 24.0): Output video frame rate.
  • —translation_speed (float, optional, defaults to 0.05): Per-frame camera translation speed for W/A/S/D actions.
  • —rotation_speed_deg (float, optional, defaults to 0.5): Per-frame camera yaw speed in degrees for J/L actions.
  • —pitch_speed_deg (float, optional, defaults to 0.2): Per-frame camera pitch speed in degrees for I/K actions.
  • —pitch_limit_deg (float, optional, defaults to 60.0): Maximum absolute camera pitch in degrees.
  • —fov_deg (float, optional, defaults to 70.0): Horizontal camera field of view in degrees.
  • —num_videos_per_prompt (int, optional, defaults to 1): The number of images to generate per prompt.
  • —image (Image | list): Reference image(s) for denoising. Can be a single image or list of images.
  • —image_crf (int, optional): H.264 CRF used to re-compress the conditioning image before VAE encode, matching the compression the model was trained against. None (default) resolves from the text-encoder generation (33 through LTX-2.3, 18 for LTX-2.5). Pass 0 to skip re-compression. Requires a PIL.Image.Image when re-compression runs.
  • —generator (Generator, optional): Torch generator for deterministic generation.
  • —num_inference_steps (int, optional, defaults to 30): The number of denoising steps.
  • —timesteps (Tensor, optional): Timesteps for the denoising process.
  • —sigmas (list, optional): Custom sigmas for the denoising process.
  • —latents (Tensor, optional): Pre-generated noisy latents for image generation.
  • —noise_scale (float, optional): Interpolation factor between random noise and any provided latents. None (default) resolves to 0.0, which keeps the provided latents.
  • —audio_latents (Tensor, optional): Optional pre-encoded audio latents; random noise is used when not provided.
  • —**denoiser_input_fields (None, optional): conditional model inputs for the denoiser: e.g. promptembeds, negativeprompt_embeds, etc.
  • —use_cross_timestep (bool, optional, defaults to True): Whether to condition the transformer on a separate per-token cross timestep (LTX-2.3+).
  • —attention_kwargs (dict, optional): Additional kwargs for attention processors.
  • —output_type (str, optional, defaults to pil): Output format: 'pil', 'np', 'pt'.
  • —decode_timestep (None, optional, defaults to 0.0): The timestep at which the VAE decodes the final latents.
  • —decode_noise_scale (None, optional): Noise interpolation factor applied to the latents at the decode timestep.
  • —vae_tiling (bool, optional, defaults to True): Enable spatial and temporal VAE decoding tiles to reduce peak memory usage.
  • —vae_tile_size (int, optional, defaults to 512): Spatial tile long-side size in pixels; the short side follows the video aspect ratio.
  • —vae_tile_overlap (int, optional, defaults to 64): Spatial tile overlap in pixels.
  • —vae_temporal_tile_size (int, optional, defaults to 64): Temporal tile size in sample frames, excluding the causal boundary frame.
  • —vae_temporal_tile_overlap (int, optional, defaults to 24): Temporal tile overlap in sample frames.

Outputs:

  • —videos (list): The generated videos.
  • —audio (Tensor): The generated audio waveform.