MATLOWAI/MiniMax-H3-Motion-Adapter

1---2license: mit3base_model: MiniMaxAI/MiniMax-H34base_model_relation: adapter5tags:6  - minimax-h37  - lora8  - video9  - comfyui10  - motion11library_name: diffusers12pipeline_tag: image-to-video13---14 15# MiniMax-H3 motion adapter (pilot, r16)16 17A rank-16 LoRA that makes the de-rope pass in18[ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes) behave better19on fast motion. It needs that pass; it is not a general-purpose motion LoRA.20 21## What it looks like22 23One 5 s clip, four ways. Every panel has its numbers burned in.24 25<video controls muted playsinline preload="metadata" width="100%"26       poster="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad_poster.jpg">27  <source src="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad.mp4" type="video/mp4">28  <a href="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad.mp4">Download the comparison clip (mp4, 2.9 MB)</a>29</video>30 31|   | left | right |32|---|---|---|33| **top** | the source plate, straight out of H3 | the whole clip re-generated on a stretched clock, base model, 139.9 s |34| **bottom** | only the fast span re-generated, no adapter, inject 0.45. 63.6 s, alternation 0.370, rate 1.416 | the same span **with this adapter** at strength 0.75, inject 0.30. 49.9 s, alternation **0.134**, rate **1.011** |35 36Watch the bottom two panels, not the stills. Without the adapter the fighters37advance and snap, advance and snap, frame to frame. With it they keep moving.38Rate 1.011 means the motion comes out at the speed the shot called for; 1.41639means the pass invented about 40 percent more movement than it should have.40 41Two things this exhibit is not. It is not the adapter on its own: the stretched42clock comes from the [ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes)43de-rope pass, and the adapter only changes how the model behaves once that pass44has stretched it. Load this file on a plain text-to-video render and you get the45costs listed below with none of the benefit. And the settings shown here are460.75 / inject 0.30, the lower-invention cell; see "Settings that work" for when47to prefer strength 1.0.48 49## Reproduce the four-tile fight example50 51The original source plate, three API workflows, a runner, and a freshly verified52adapter result are now in [examples/fight_quad](examples/fight_quad/README.md).53For the normal ComfyUI canvas, download [the winner workflow](examples/fight_quad/adapter.canvas.json)54or [the complete four-tile workflow](examples/fight_quad/fight_quad.canvas.json),55then drag the JSON into ComfyUI.56 57Start with the included **seedhunt_20260963_00001_.mp4** source; the top-left tile58is that original video. The other three tiles are the full-clip base pass,59windowed v3.1 without the adapter, and the adapter at **0.75 / inject 0.30**.60 61[Download the source plate](examples/fight_quad/seedhunt_20260963_00001_.mp4) ·62[Download the verified adapter result](examples/fight_quad/winner_verified_20260921.mp4) ·63[Reproduction instructions](examples/fight_quad/README.md) ·64[Download the complete example ZIP](examples/fight_quad_example.zip)65 66The September 21 verification completed in **65.4 s** and decoded successfully67(124 frames, 1152 × 640, 24 fps). This is a fresh functional check, not a new68matched-cache benchmark. The original comparison's 139.9 / 63.6 / 49.9 s labels69remain historical measurements. Both supplied MP4s contain their API graph in70`prompt` metadata; separate JSON files are included for convenient reuse.71 72## How it is actually wired, and what the adapter is not responsible for73 74Same clip, same seed, same graph in all four tiles. An anime sword ring, which75is the content type it helps most.76 77<video controls muted playsinline preload="metadata" width="100%"78       poster="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad_poster.jpg">79  <source src="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad.mp4" type="video/mp4">80  <a href="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad.mp4">Download the comparison clip (mp4, 2.7 MB)</a>81</video>82 83|   | left | right |84|---|---|---|85| **top** | the source plate | the burst window re-generated at denoise 0.70 with **nothing anchoring it**, so it drifts off the plate and jumps at both splices |86| **bottom** | the same graph with the window's **first and last frames pinned**. Two extra wires, no extra time (10m41s either way): jitter 0.103 to **0.006**, entry jank 2.05 to 1.58, invented objects 109 to **60** | pins **plus this adapter** at 0.75, one `LoraLoaderModelOnly`: alternation 0.129 to **0.093**, objects 60 to 54, for 39 s more wall time |87 88The order matters, and it is the opposite of what people expect. **The pins do89most of the work**, and they are free: pinning the window's boundary frames to90the plate frames they must butt against is what stops the splice jump, adapter91or no adapter. The adapter then converts spare capacity into smoothness on top92of that. If your seams are jumping, pin first and only then reach for this file.93 94The same tile also shows the cost honestly: warm colour goes 1.04 to -1.59 with95the adapter on. That is the colour muting described below, visible in the one96exhibit where everything else improves.97 98Play to the end for a card listing every setting: window, hold map, denoise,99sampler, seed, quantisation and wall time per tile. (It labels the LoRA by its100training name, `p4_pilot_k100.safetensors`; that is this file, published here as101`minimax_h3_motion_adapter_pilot_r16.safetensors`.)102 103More before/after clips, including the two cases where it makes things worse,104are on the [demo page](https://matlowai.github.io/ComfyUI-MAINodes/#adapter).105 106## About this release107 108It is a pilot: rank 16, trained for a few hours on10938 clips we generated ourselves. It works, it is measurable, and it has110clear rough edges listed below. We are publishing it as an intermediate111release while we try something considerably more ambitious that may or may112not work (an all-in-one adapter that internalises the whole two-pass113pipeline). If that lands it will replace this file; if it does not, this114one still does a useful job. Part of that work is finding out whether an115adapter can be made to actually assign more compute to the burst on its116own (a variable-rate attention behaviour learned inside the model, rather117than the nodes stretching the clock for it); this pilot does not do that,118see the mechanics note below.119 120## What it does121 122MAINodes' Motion Lab regenerates the fast span of a clip on a stretched123clock (the "de-rope") and recovers it back to real time. That pass has a124known bad habit: on very fast motion the base model over-produces, and the125result alternates between advance and snap frame to frame. This adapter126was trained on that exact task (frames held out of the middle of a burst,127the model asked to fill them back in), and it teaches the base model to128spend the extra clock on smoothness instead of invention. To be precise129about the mechanics: the LoRA is a plain rank-16 weight delta and adds no130compute and needs no custom node. The extra clock comes from the nodes131(the de-rope stretches the burst into more latent tokens); the LoRA132changes how the model behaves on that stretched clock, because it was133trained in that exact situation.134 135Measured on four clips the adapter never saw during training (a fight,136an anime sword ring, a character swap, a dialogue close-up), against the137same graph without the adapter and against a same-seed noise floor:138 139- advance/snap alternation improved on 4 of 4 clips (4.6 to 6.1 times the140  floor)141- frame-to-frame over-production dropped from 1.76 to 2.93 down to 1.09142  to 1.23 on all four143- blind operator picks, adapter vs no adapter, same seed: 7 for, 0144  against, 1 no preference145- it transfers to the Ref2VA checkpoint (span, alternation and jitter146  wins of the same size on the two clips we tried), so one file serves147  both fl2va and ref2va graphs148- one end-to-end run of the shipped graph on a scene it had never seen149  (the fight choreography moved to a neon rooftop, same seed for both150  arms): alternation 0.24 to 0.19, jitter 0.09 to 0.04, fewer invented151  objects, and it kept the plate's setting where the no-adapter pass152  rewrote the signage. One clip, and each graph at its own shipped inject153  (0.70 vs 0.45), so read it as a smoke test, not a study154 155Where it is not great (also measured):156 157- it costs about 1 dB of anchor fidelity on native keyframes and it158  over-corrects calm chains (it smooths things that did not need159  smoothing); keep it off on calm content160- it mutes strong colour and thins particle effects. A prismatic creature161  came out as a plain calico on the Ref2VA checkpoint (mean saturation 46162  to 38 at denoise 0.40, 54 to 40 at 0.70) and again on the fl2va163  checkpoint when the whole clip was de-roped at 3x. Worst on Ref2VA and164  on full-clip passes; a windowed de-rope on the same creature kept its165  colour. If the subject is the colour, lower the strength or accept the166  trade167- it is anime-heavy in effect: on the one very fast anime clip we168  measured it wins clearly at every setting; on live-action character169  work it needs the lower inject values below170 171## Settings that work172 173- Strength 1.0. Strength and inject turned out to be the same dial. 1.0174  wins on alternation in every paired cell we ran (8 of 8), while 0.75175  invents fewer objects (7 of 8), so 1.0 is the smoothness setting and176  0.75 the lower-invention one; the old 0.75 to 0.80 band was177  in-distribution tuning on one fight plate and does not hold elsewhere.178- Inject is the dial. Starting points: 0.45 for character or dialogue179  where the base model over-produces hard; 0.30 where identity or props180  are the deliverable and the base is only mildly inflated; on very fast181  anime it does not matter (every setting wins).182- Keep the tail guide on. Turning it off costs identity about 3 times the183  floor across seeds and checkpoints, adapter or not.184- Apply it to the de-rope pass only, not to the first (text-to-video)185  pass.186 187## Use in ComfyUI188 189Put the file in `models/loras/minimax_h3/`. Load it with a stock190`LoraLoaderModelOnly` at strength 1.0 on the model that feeds the de-rope191pass. A ready graph is in ComfyUI-MAINodes:192`examples/motion_pipeline_adapter_api.json` (API format, loads through the193API and through the frontend's API import). The stock194`LoraLoaderModelOnly` is fine for this file; the key mismatch that bites195turbo LoRAs does not apply here.196 197## Training facts198 199- Base: MiniMax-H3 (fl2va), rank 16, alpha 16, 375 steps, lr 1e-4,200  adamw8bit, bf16, about 4.6 h on one GPU201- Data: 38 holdout-infilling items cut from 15 clips we generated with202  MiniMax-H3 (physics scenes, fights, chains, a dialogue plate); no real203  footage, no third-party footage204- Task: hold out the frames inside a motion burst, keep the rest as clean205  context (kept_label 1.0), predict the held tokens206- Trainer: our ai-toolkit extension for H3 (span-aware packing verified207  bit-exact against ComfyUI)208 209## The temporal-expansion ladder (new, 2026-08-24)210 211The "considerably more ambitious" follow-up mentioned above has produced its212first shippable piece. The pilot learned its task from targets the model213generated itself; these checkpoints retrain the same recipe against real214intermediate frames, and every training step is published so you can pick215your own point on the curve.216 217**What changed.** Same rank-16, attention-only construction, warm-started218from the pilot. The training targets are now real: windows cut from the219public GOPRO_Large deblurring dataset (240 fps source), decimated to the220working frame rate and re-expanded, so the held tokens are scored against221frames a real camera actually captured rather than an interpolation. The222world clock in training is truthful (each token labelled with the real time223it spans).224 225**Measured, on scenes the adapter never saw** (four camera shoots fully226disjoint from training, held-token error in latent space, lower is better;227the no-adapter baseline is 1.23 and the plain interpolation the pass starts228from scores 0.26):229 230| step | 25 | 100 | 175 | 250 | 375 |231|---|---|---|---|---|---|232| error | 0.158 | 0.133 | 0.116 | 0.105 | 0.086 |233 234The curve never turns: more steps keep measuring better, through the last235checkpoint. And here is the honest part: **the numbers and the eye disagree236about where to stop.** Step 100 was the pick in playback: fast motion comes237out smooth and the clip still feels like itself. Step 375 measures best by238a wide margin, but in side-by-side viewing it can read over-smooth, as if239the motion had been graded. On a static-content probe the family does not240invent motion (it lands below the interpolation floor rather than above241it), so the disagreement is about feel, not correctness. We publish every24225th step so you can find your own point; if you only take one file, take243`warm100`.244 245**Watch instead of reading.** Six arms of one clip, every panel with its246settings and wall time burned in; the bottom-right panel is the full247de-rope end to end for cost comparison:248 249<video controls muted playsinline preload="metadata" width="100%"250       poster="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/temporal_expansion_2x3_tile_poster.jpg">251  <source src="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/temporal_expansion_2x3_tile.mp4" type="video/mp4">252  <a href="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/temporal_expansion_2x3_tile.mp4">Download the comparison tile (mp4, 11 MB)</a>253</video>254 255**Use.** Exactly like the pilot: drop a file from `temporal_expansion/`256into `models/loras/minimax_h3/`, load with a stock `LoraLoaderModelOnly`257at strength 1.0 on the model that feeds the de-rope or windowed-insertion258pass. Not for the first text-to-video pass.259 260**Files.** `temporal_expansion/minimax_h3_temporal_expansion_warm100_r16.safetensors`261(the playback pick, sha256 cbb23ad4...) and262`minimax_h3_temporal_expansion_step025_r16.safetensors` through263`step375_r16.safetensors`, every 25 steps of the same run (step375 sha25626428f3d41d...). 63 MB each.265 266**Training facts.** Warm-started from the pilot above; 375 steps, lr 1e-4,267adamw8bit, bf16; 48 samples derived from GOPRO_Large (Nah et al., the268public deblurring benchmark) plus the pilot's self-generated set for the269difficulty gate; the trainer's span-aware packing verified bit-exact270against ComfyUI. Unlike the pilot, this data includes real footage; the271source dataset is public and widely used for research, and the derivation272(decimate, re-expand, score held tokens) is described so it can be273reproduced or swapped for your own footage.274 275## Base model and licence276 277This is a LoRA for [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3)278by MiniMax; it does nothing without their base weights and is trained on279clips generated with them. The adapter weights and the graph here are280released under the MIT licence (`LICENSE`); use of the base model, and so281of this adapter with it, is subject to the MiniMax H3 Community License282Agreement, included as `LICENSE-MiniMax-H3` (copied from the MiniMax-H3283repository at commit 42ed227e). Thanks to MiniMax for releasing H3.284 285## Files286 287- `minimax_h3_motion_adapter_pilot_r16.safetensors` (63 MB)288- `motion_pipeline_adapter_api.json` (the same graph as in the node pack)289- `LICENSE` (this repository, MIT) and `LICENSE-MiniMax-H3` (the base290  model's community licence)291 292Examples and before/after clips, including the colour-muting case, are on293the demo page: https://matlowai.github.io/ComfyUI-MAINodes/#adapter294The node pack, the example graph and the tuning notes are at295https://github.com/matlowai/ComfyUI-MAINodes (see README, "The motion296adapter (pilot)", and TUNING.md).297