joeygambino/joyai-echo-ltx23-echoVid-ltxAud-surgical

The VIDEOONLY distill LoRAs now have their own repo: joeygambino/ltx23-videoonly-distill-loras. They still ship here for continuity, but the dedicated repo is where they are documented (measurements, strength guidance, the "small strip is the better strip" result) and where updates will land.

JoyAI-Echo x LTX-2.3 — echoVid + ltxAud surgical merge

Nodes, workflow, issues and releases: https://github.com/jlucasmcrell/joyai-echo-multishot-patch All models: https://huggingface.co/joeygambino

A word on expectations: the multishot workflow this feeds is powerful but not one-click - first renders usually take some tuning to your VRAM, system RAM and model build. The workflow repo's INSTRUCTIONS.md covers setup and known failure modes, and I am happy to troubleshoot: open a discussion and I will answer.

JoyAI-Echo's video/conditioning branch married to LTX-2.3-distilled-1.1's audio branch.

Speed: turn on fp8_scaled_mm

If your renders crawl in the JoyAI-Echo pack, this is almost always why. It is off by default.

fp8_scaled_mm stores the DiT linears as float8_e4m3fn and computes the matmuls natively via torch._scaled_mm on RTX 40/50-series — unlike fp8_transformer there is no per-layer upcast tax.

The short version: feed it bf16 or INT8 and turn the toggle on. The fp8 file is for running without the toggles.

(Under stock ComfyUI rather than the pack, the INT8 build instead runs on the native W8A8 loader — a different path with its own speed profile.)

Render at 24 fps — voices and accents (2026-07-30)

LTX-2.3's joint audio-video prior is 24 fps-native, and render fps turns out to be a hidden accent dial: at 25 fps the same prompt and seed render non-rhotic southern British, at 30 fps broad Australian — and any off-24 fps setting overrides accent wording in the prompt entirely (a few conditioning tokens cannot outvote a geometric signal present in every attention op at every step). Verified by A/B on identical configs: 24 fps = rhotic General American at 5/5 reviewer confidence, same text and seed.

Keep video_fps at 24 and state the accent in the positive prompt ("in a casual American accent"). Flip side: if you want an authentic British or Australian voice, rendering that scene at 25/30 fps is more consistent than any accent wording.

Demo

<video controls src="https://huggingface.co/joeygambino/joyai-echo-ltx23-echoVid-ltxAud-surgical/resolve/main/demothewitness_fp8.mp4"></video>

THE WITNESS - five shots, five locations, one woman, one voice; nothing carries her between shots except one repeated sentence and the cross-shot memory bank. Rendered with the bf16 checkpoint in this repo under the fp8 scaled-mm compute path. AI-generated video and audio.

Two strong models, each with one disqualifying weakness for multi-shot

character work:

Neither is shippable alone for a talking-character piece: Echo keeps the same

person across ten shots but sounds synthetic; LTX sounds like a real performer

but casts "three different actresses" across three shots.

This merge takes the half that works from each.

What was merged

Embedded in the file's own __metadata__ as rift_built_from:

SURGICAL MERGE: video/conditioning+av_ca-modulation from JoyAI-Echo-release;



audio + both cross-attns from distilled-1.1

• —From JoyAI-Echo release — video branch, conditioning path, and the

audio-video cross-attention modulation. This is what carries the slot-paired

memory bank, so face/wardrobe identity survives across shots.

• —From LTX-2.3 distilled-1.1 — the audio branch and both cross-attention

stacks. This is what produces the natural voice and lip-sync.

Architecture is unchanged (AVTransformer3DModel, 48 layers); this is a

tensor-level graft, not a retrain or a weight-average.

Files

Both are full checkpoints — DiT + video VAE + audio VAE + vocoder +

text connectors — so they drop into a JoyEcho/LTX-2.3 graph as a single

model_file with no companion checkpoint needed.

All 5,947 tensors are present, **including the four

text_embedding_projection.* tensors**, so each file loads as a single

self-contained artifact.

That last point is worth knowing if you also run GGUF quants of this family.

LTX-2.3 ships those four tensors as a separate ~2.2 GB component file

rather than inside the DiT, while JoyAI-Echo bundles them in — the two

projects package the same weights differently, and they are byte-identical

between the two. Some JoyAI-Echo GGUF conversions follow the LTX layout and

leave them out, which is a reasonable thing for a converter to do, but if

nothing in your graph then loads the companion file those tensors get built

random-initialised. The symptoms are subtle enough to be misdiagnosed as a bad

prompt — robotic voice, the model reading prompt text aloud, identity resetting

between shots — and load logs look clean either way, because they sit outside

the swept path. Verify any checkpoint with:

from safetensors import safe_open



with safe_open("model.safetensors", framework="pt") as f:



    proj = [k for k in f.keys() if "text_embedding_projection" in k]



print(len(proj), "expected 4")

Observed behaviour

Used in production for multi-shot narrated video (10 shots × ~13 s, one

recurring character). Against the same script and seed:

• —Voice: independent blind review scored it **"unquestionably a real recorded

human"**, with no drift in timbre or accent across all ten shots. The

JoyAI-Echo parent's best result on the same harness was 6–7/10 ("performed"

emotion, metallic sibilance).

• —Lip sync: 9.5–10/10 per shot, phoneme-accurate, no progressive drift.

• —Identity: a single character held across ten completely different locations

with a byte-identical identity sentence per shot and memory_max_size=7.

• —Picture: read as a real camera recording rather than AI, with correctly

formed hands and realistic eye reflection.

Two-character single-frame shots also hold, given stable ID_A/ID_B labels

and contrast on several axes at once (age band, hair, build, clothing colour).

Talking heads in plain core-node workflows (measured 2026-07-25)

Scope note for the numbers above: they were earned under the JoyAI-Echo pack's production recipe — slow, deliberate lines (~1.1 words/sec), head trims, and memory-bank conditioning.

At conversational speech rates (~2.5–3 words/sec) in a stock-sampler I2V talking-head path (LTXVAddGuide + LTXVReferenceAudio + the TalkVid ID-LoRA, 25 fps), this merge articulates the mouth measurably less than stock ltx-2.3-22b-distilled-1.1: mouth-region motion 4.8–5.2 vs 9.3 on the same seed, prompt, and settings. Rewiring experiments (reverting the cross-modal modulation to stock; moving the full cross-modal apparatus Echo-ward) did not close the gap — the ceiling lives in the echoVid video-attention weights themselves, and this merge is already the best arrangement of its parts (every alternative tested scored equal or worse).

Recommendation: for reference-voice talking heads in plain core-node workflows (e.g. the Multishot Lite workflow), use stock ltx-2.3-22b-distilled-1.1 — it articulates roughly twice as hard. Use this merge where it wins: identity-held, multi-shot narrative under the pack's recipe, where the pacing above keeps sync scoring 9.5+.

Update (later 2026-07-25): the TALKINGHEAD variant — articulation fix, baked in

For reference-voice talking heads at conversational speech rates, use the TALKINGHEAD builds instead of the base merge — no extra nodes, no LoRA to wire:

• —ltx23_echoVid-ltxAud_surgical_talkinghead_bf16.safetensors
• —ltx23_echoVid-ltxAud_surgical_talkinghead_fp8.safetensors

They are the base merge with the c72 video-only distilled LoRA (below) baked in at 0.5 — weight-equivalence verified per-module against fp32 compute. Measured across two independent seed draws at default settings: shot-start articulation 4.8 → 6.9–7.5 (stock distilled-1.1 = 9.3) with envelope timing preserved (0.23–0.27 vs the 0.25 no-LoRA baseline; the earlier full-rank bake traded timing down to ~0.09 for a little more raw motion — that is why this bake uses the truncated strip). Face untouched.

Which file do I want?

The base merge stays the pack's model: at the pack's pacing (~1.1 words/sec, head trims, memory bank) its sync already scores 9.5+, and the baked boost is measured to overdrive chained shots that open on a moving guide frame (second-shot envelope correlation ~0.40 → ~0.20). That tradeoff is why this is a separate file and not an update to the base merge.

Update (2026-07-25, latest): the e50 two-dose rebuild

ltx23_echoVid-ltxAud_surgical_e50_* attacks the same articulation deficit as the TALKINGHEAD builds, but from the weights rather than by baking a LoRA on top.

Scope: this is a fix for the reference-audio path, not a general lip-sync repair. The deficit appears when a character is driven from a supplied voice clip (LTXVAddGuide + LTXVReferenceAudio + the TalkVid ID-LoRA) at conversational speech rates (~2.5–3 words/sec). Under the pack's own production recipe — slower, deliberate delivery plus memory-bank conditioning — the base merge already scores 9.5+ on sync and is not outclassed by e50. If you are not driving from reference audio, you do not need this build.

The video side is reconstructed from two independent doses:

video          = dev + 1.0 x (distilled-1.1 - dev) + 0.5 x (JoyAI-Echo - dev)
tables/adalns  = dev + 0.3 x (distilled-1.1 - dev) + 0.5 x (JoyAI-Echo - dev)
audio + cross-modal = stock distilled-1.1   (unchanged, so TalkVid still works)

Why this shape. JoyAI-Echo is a dev-lineage finetune — its training fingerprint matches ltx-2.3-22b-dev, not distilled-1.1 — so on the 8-step distilled sigma ladder its video stack is effectively under-distilled. That surfaces as under-articulated mouths and smeared fine motion when the mouth is being driven hard by reference audio at speed; at slower delivery it is not the limiting factor. Adding the full distillation direction back restores articulation; holding Echo's own finetune at 0.5 recovers stock's crisper viseme formation, which is most visible on rounded vowels (on the test line, "cloned" forms a real /oʊ/ with visible tongue movement instead of a flattened closure). The modulation tables take a lighter distill dose because at full strength they produce specular, over-shiny skin.

Both doses were chosen by rendering and eye, not by theory: a full-strength distill correction reads rubbery/exaggerated, stock alone reads stiff, and 0.5 was picked over 0.6 and 0.7 on viseme shape.

Known trade. Echo's finetune is what suppresses the occasional robotic voice and drives the pack's audio memory bank. At 0.5 you keep half of it. For long multishot runs through the pack, the base merge remains the safe choice.

e50 vs TALKINGHEAD. Two different routes to the same problem; they have not been benchmarked head-to-head at equal settings. e50 is the newer and the one in current use here. If you already have a tuned TALKINGHEAD setup, there is no measured reason to switch.

Shot length: this is a nodes issue, not a weights issue. Through the JoyAI-Echo pack, lip sync decays and visibly breaks about 9.6 s into every shot — the wrapper's video RoPE clock is hardcoded to 24 fps while audio RoPE is in true seconds, so a 25 fps render drifts ~4 %/s. It reads as a model limitation and is not one: no LTX-2.3 or Echo checkpoint caps shot length (temporal RoPE range is 20 s, identical across Echo, dev and distilled-1.1), and no merge can fix it because the bug is in coordinate math. The multishot patch stamps the real render fps onto the generator (Bug fix #0); with it, 60–105 s multishot masters hold sync.

Quantised e50 builds (Q80 / Q50 / Q4_0) are in the GGUF repo.

The underlying fix, as a LoRA (for strength-tuners)

Two strips are available; the small one is the better one:

• —Recommended: [`ltx-2.3-22b-distilled-lora-c72_VIDEOONLY.safetensors`](./ltx-2.3-22b-distilled-lora-c72_VIDEOONLY.safetensors) (390MB, rank-72 truncation). At 0.5 under the TalkVid ID-LoRA it keeps the articulation boost AND times better at default identity guidance — shot-start envelope correlation 0.32 vs 0.10 for the full-rank strip, replicated across independent seeds. Rank truncation appears to drop motion-texture components that decorrelate from the audio envelope.
• —ltx-2.3-22b-distilled-lora-384-1.1_VIDEOONLY.safetensors (3.76GB, full-rank). Slightly more raw motion; measurably worse envelope timing at default guidance. Kept for completeness and strength-sweepers.

The deficit above decomposes into two independent causes, and the amplitude half is fixed by an adapter shipped in this repo: `ltx-2.3-22b-distilled-lora-384-1.1_VIDEOONLY.safetensors`.

Why it works: this merge's video side is under-distilled — dev-flavoured attention driven by an 8-step distilled schedule undersamples, and mouth articulation is the first casualty. The LoRA is the dev→distilled delta (rank 384) with all audio and cross-modal keys stripped (1174 video keys kept), so it restores the missing distillation direction without touching the pathway the TalkVid ID-LoRA adapts.

Measured, same seed/prompt/settings (mouth-region motion / audio-envelope correlation): shot-start articulation goes from 4.8 → 7.6–8.0 (stock distilled-1.1 = 9.3) at strength 0.5, chained under the TalkVid ID-LoRA, with no face restyling.

Known costs, measured: in chained multi-shot use the boost overdrives shots that open on a moving guide frame (second-shot envelope correlation drops from ~0.40 to ~0.20) — for multi-shot work, sweep strength down (0.2–0.35) or apply it per-shot. Do NOT use the full (non-VIDEOONLY) rank-384 distilled LoRA under the TalkVid ID-LoRA: its audio/cross-modal keys fight the ID-LoRA and envelope timing collapses to zero.

Separate timing note: identity_guidance_scale amplifies adherence to the voice reference. When the reference clip's words differ from the spoken line (always true for the first shot), that amplification fights the script's timing — measured 0.25 → 0.43–0.54 envelope correlation by setting the FIRST shot's scale to 0–1.5. But in chained workflows the second shot references the first shot's own audio, and an unguided first shot produces a worse chain reference (second-shot timing drops). Single-shot talking heads: lower the scale. Chained shots: keep 3.0 on the chain, tune only shot 1, and judge the whole clip.

Usage notes

• —Requires the memory-multishot path to get the identity benefit — a

plain single-shot graph will not exercise the memory bank.

• —sequential_offload is effectively mandatory: 43 GB of bf16 weights will

not fit a 32 GB card unquantised. The fp8 build at 23.4 GB is the practical

choice for 24 GB hardware.

• —Keep negative-steering gentle. negative_scale_video above ~1.0 rotates the

conditioning hard enough to scramble per-shot scene content — every shot

collapses onto shot 1's composition while the audio advances correctly.

0.5–0.8 is the working band.

• —Prompts should state the accent affirmatively ("in a casual American

accent"). The model's default drifts British.

Provenance and license

Derived from:

• —Lightricks/LTX-2.3 — © Lightricks Ltd.

• —jd-opensource/JoyAI-Echo — JD Joy Future Academy

Distributed under the LTX-2 Community License Agreement, a complete copy of

which is included in this repository and embedded in each file's

__metadata__.license. Per §3 of that license this derivative is distributed

exclusively under those terms, and the modification is disclosed above and in

rift_built_from.

Non-commercial. LTX-2 itself permits commercial use below $10M ARR, but the

JoyAI-Echo component is released for academic research and non-commercial use

only, so the stricter term governs this merge.

Use is subject to the LTX-2 Acceptable Use Policy (Attachment A of the

license), which among other things requires that generated content be

disclosed as machine-generated.

No affiliation with or endorsement by Lightricks or JD is claimed or implied.

One upstream metadata field, encrypted_wandb_properties (an encrypted

training-run blob inherited from the original weights), was removed from both

files. Nothing else in the header was altered and no tensor data was touched.

Support

Everything here is free and stays free. If it saved you time, you can sponsor me on GitHub, buy me a coffee, or support me on Liberapay.