CoolFace
Datasetpublic

cs-mshah/layergen-eval-latents

LayerGen — Eval-Set Latents (VAE latents + baked text embeddings) Pre-encoded evaluation-set inputs for the LayerGen layer-decomposition / harmonization models, so inference can run anywhere (off-AIP) without the raw video → VAE-encode → umT5-encode pipeline. Each *.parquet is one clip and is fully self-contained: column group contents {composite,mask,fg,bg}_latent_bytes (+ _shape, _dtype) 4-stream Wan-VAE latents, 81f/21 latent-T, fp16, [16,21,60,104]… See the full description on the dataset page: https://huggingface.co/datasets/cs-mshah/layergen-eval-latents.

sourceHugging Facemitupdated 1mo agoView on Hugging Face
0likes861downloads
Dataset Card

LayerGen — Eval-Set Latents (VAE latents + baked text embeddings)

Pre-encoded evaluation-set inputs for the LayerGen layer-decomposition / harmonization models, so inference can run anywhere (off-AIP) without the raw video → VAE-encode → umT5-encode pipeline.

Each *.parquet is one clip and is fully self-contained:

column groupcontents
{composite,mask,fg,bg}_latent_bytes (+ _shape, _dtype)4-stream Wan-VAE latents, 81f/21 latent-T, fp16, [16,21,60,104]
text_embedding_bytes (+ _shape, _dtype)baked umT5 edit-instruction embedding (Wan text encoder, 512-token)
caption, source, id, width, height, num_frames, fpsmetadata

Because the umT5 embedding is baked in, no text encoder is needed at inference — read with data_format: latents and the per-clip text_embedding comes straight from the parquet.

Pools (latest editcap captioning scheme)

poolclipstasknotes
_val_latents_81f_editcap_dec_combined/all678decomposefull eval: RORD50 / PROVE-M/H / VOR-Eval/Wild / ROSE-Bench + synthetic
_val_latents_81f_editcap_har_combined/all458harmonizefull harmonize eval
_val_latents_81f_editcap_dec_pvix/all76decomposePVIX (no-GT)
_val_latents_xcompv2_editcap_har/all75harmonizecross-composition (harmonize-hard-v1)
prompt_fix_pool/all306decomposeCFG expt — 34 clips × 3 prompt-fixes × 3 seeds (no-CFG)
prompt_fix_pool_cfg/all171decomposeCFG expt — CFG-2 subset
prompt_fix_pool_har/all180harmonizeCFG expt — 20 xcomp clips × 3 × 3 (CFG + no-CFG)

Crop to 41f/61f at load via num_latent_t (11 / 16 / 21). Type-4 harmonize checkpoints require FASTVIDEO_FG_CORRUPT_TYPE_ID=4; combined checkpoints need COMBINED_INFER=1.

Off-AIP inference (recipe)

python
from huggingface_hub import snapshot_download
# pull one pool locally
snapshot_download("cs-mshah/layergen-eval-latents", repo_type="dataset",
                  allow_patterns="_val_latents_81f_editcap_dec_combined/**",
                  local_dir="evals")
# then point the inference config's validation.data_path at
#   evals/_val_latents_81f_editcap_dec_combined/all   (data_format: latents)

DiT checkpoints are public under `cs-mshah`; decode uses the public Wan2.2-14B VAE. Companion output-viewer datasets: cs-mshah/layergen-evals (+ -baselines).