infosave

/

MiniMax-H3-Turbo-cmf

like 11

MiniMax H3 Realism People LoRA
Trigger word: r34l1sm

A LoRA adapter for MiniMax H3 specialized in realistic people: faces that hold up in close-up, natural skin texture, believable expressions and gestures, film-style lighting and documentary camera movement.

https://huggingface.co/fal/MiniMax-H3-Realism-People-LoRA

LoRA Spatial & Physics (MiniMax-H3)

LoRA, a spatial reasoning and physical logic model, is trained on MiniMax-H3 (Comfy-Org/MiniMax-H3) and enables the model to learn the spatial relationships and physical motions of pure objects (collision, stacking, falling, occlusion, etc.).

https://huggingface.co/Jojocodex/minimax-h3-spatial-physics-lora

TX-2.5 22B IC-LoRA Pixel Spatial Upscaler

(posting here cause this may be an interesting workflow ... using LTX 2.5 lora to enhance mini max outputs)

https://huggingface.co/Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler

How It Works

The model is an in-context LoRA: instead of taking only a text prompt, it also consumes a reference video in context alongside the noisy latents being denoised. During training, the reference is a downscaled version of the target clip, so the model learns to map a low-resolution reference onto a higher-resolution rendering of the same scene.

At inference the reference is your low-resolution clip. The model does not upsample it pixel-wise; it re-renders the scene at the higher resolution, inventing plausible fine detail (texture, edges, micro-contrast) that was never present in the reference. Composition, motion and subject identity are carried over from the reference, while high-frequency detail is generated.

Another upscaler for mini max

https://github.com/LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler

https://huggingface.co/mvp-lab/MiniMax-H3-RAVEN-Streaming-LoRA

by Claude
Transforms MiniMaxAI/MiniMax-H3 into a "causal streaming generator." Instead of denoising the entire video clip bidirectionally as a whole, it generates content by processing in chunks, extrapolating the next part from the already generated content. This design enables sequential and more real-time-like generation.

Thank you for the collection — this is the most useful thread on the repo. Triaged honestly, one by one:

• fal Realism People — works today. 0.5.92 runs it at runtime: rank 32, 104 branches, all of them bind (it trains attention only). Details and cost in #8; A/B clips are in the repo under samples/lora_ab_*.mp4.
• Jojocodex spatial-physics — 516 tensors, and 50 of them sit on adaln_proj.linear. The attention and MLP halves bind; the adaLN half needs baking (animate-pack --lora --time-embedder), because this container holds the modulation as a curve. It will run and tell you which branches sat out.
• mvp-lab RAVEN streaming — the interesting one for your 24 GB problem, and it is more than weights: it changes the sampler from denoising the whole clip bidirectionally to causal chunks, which is exactly what stops the activation cache growing with frame count. The adapter also touches adaln_proj, time_embedder, condition_proj and both patch projections — so it wants a pack-time fold plus a new sampling loop. On the list, and it is the right fix for long clips on a Mac.
• LBH latent upscaler — I read its header: 345 M parameters, a 3-D conv net over the 24-channel latent (conv_in → 6 in_blocks → 6 out_blocks → conv_out, with a timestep embedding). That is very portable — the same shape as the LTX latent upscaler already in the engine — and it is the honest answer to "next quality jump on 24 GB": render small, upscale the latent, refine a couple of steps, skip the 5 B-parameter VAE round trip entirely.
• Lightricks IC-LoRA upscaler — needs in-context video conditioning: the reference clip rides in the sequence beside the noisy latents. We have that machinery for reference stills (--ref on ltx-video), not clips. Meanwhile LTX already has a native two-stage path: cortiq ltx-video --two-stage renders at half resolution, upscales the latent and refines — try that before the IC-LoRA.

One thing that came out of running your first link: adapters are not uniform across depth. CMF_LORA_PROBE=1 prints each branch's contribution, and on the Realism adapter the loudest is 150× the quietest, blocks 0–2 contribute nothing it would miss, and 41 of 104 branches carry the look.

The latent upscaler is ported — 0.5.93, and it runs rather than being described:

cortiq animate model.cmf --prompt "…" --width 512 --height 288 \
  --upscale minimax_h3_latent_upscaler_3d_fp16.safetensors --upscale-by 2.0

Render small, the net resizes the latent, the VAE decodes once at the larger size — no decode → resize → encode round trip through the 5 B VAE. Parity against the node's own torch module on the same weights: worst 6.7e-6, relative rms 3.9e-7.

Also from this thread's list: the spatial-physics adapter is now measured, not predicted — lora: rank 16, 208/258 branches bound; not applied: blocks…linear ×50, and the render completes. RAVEN streaming is the one I have not started: it needs a chunked causal sampler, not just the weights.

RAVEN streaming runs. The container is up: mmh3-raven-streaming-q4tp.cmf (14.88 GB, 2698 tensors, the adapter folded in at pack time — including time_embedder.proj_in/out, which a runtime --lora cannot reach because the time curve is tabulated).

cortiq animate mmh3-raven-streaming-q4tp.cmf \
  --prompt "a woman in a red raincoat walks down a neon-lit street at night" \
  --width 384 --height 256 --frames 161 --steps 4 \
  --stream-chunk 13 --stream-sink 2 --stream-window 2 --out take.avi

Chunk-causal rollout: each chunk attends to a sink of the first chunks and a sliding window of the last, and the finished frames enter at timestep 0 as clean context.

The chunk length is the knob — RTX 5090, 384×256, 41 frames in (56 out), four steps:

At five latent frames the context is not what is missing (with three chunks, sink 2 / window 2 sees every earlier one) — the chunk is simply shorter than what the adapter learned. Nine to thirteen holds.

Honest cost: streaming is ~1.7× the denoise of the ordinary path at this length, because a chunk's context rows are recomputed at each of its four steps. A KV cache over those rows removes that; it needs an attention kernel that takes external K/V, which the fused path does not have yet. Next up.

you must be a bot. this is incredible progress

I started programming on a calculator in 1994. Now I’m delighted that I can delegate a huge amount of work to my own agents, built on my own technologies 😃

good bot!

so i put all your latest threats in gemini to understand. going through them so ill have specific comments or feedback later.

but one thing is unclear. are we able to do the references and character sheets now? other than my first frame hack? there is a problem with that one as it continous the action so wastes frames. on 24gb mac min i have a frame budget of like 10 if i want to proto type stuff.

are we at feature parity with what mini max h3 provides out of the box?

also do you think there is any possible way that i can use both my 24 gb and my m1 /64gb to share the rendering load of one prompt? gemini says no. but maybe with the new loras (streaming?) something is possible?

idea is to give it 60 frames, min takes first 20 it can handle, the rest are sent to m1 /64.

gemini seems to think cause its a diffusion model it wont be possible as things have to move linearly. but maybe...just maybe?

You asked twice whether we are at parity with what H3 offers out of the box, and you deserve the list rather than another feature announcement. Here it is, honestly:

Works today

• --first-frame / --last-frame (fl2va: the picture conditions the DiT and enters the prompt as a vision block).
• --video <dir> --video-stride N — a directory of reference frames, every N-th one pinned to its own moment. This is v2v as this container can do it: the keyframe path with more than two frames. It is also the closest thing to a character sheet without the "first frame eats my resolution budget" hack you described.
• --lora at runtime (any of the three naming conventions in the wild), or folded in at pack time for the parts a runtime adapter cannot reach.
• --upscale — the latent upscaler, so you render small and decode large.
• --stream-chunk — new: the RAVEN chunk-causal path, container up as mmh3-raven-streaming-q4tp.cmf.

Not there

• Reference audio in (your --audio reference.wav). The layout already has the row kind for it and the container already carries the audio VAE's encoder half, so this is now a wiring job rather than a training one. That is a change of status since I last answered you in #3 — I said it needed training; with the encoder packed, it does not. I am not promising a date, but it is no longer the wall it was.
• Lip-sync driven by an audio track, and the reference's own video node. Those are model-side, not runtime-side.

On your two Macs — 24 GB and the M1/64 GB. Gemini told you no, and for the ordinary path it is right: a bidirectional diffusion step needs the whole clip's latent at once, so there is nothing to cut in half. The chunk-causal path changes the shape of that answer, though not into a speedup:

• A chunk only needs the finished frames of earlier chunks — a sink of the first and a sliding window of the last. So a clip can be handed from one machine to the next in sequence: the 24 GB Mac renders chunks 1..k, passes the finished latents, the 64 GB M1 continues. One clip, two machines, no shared memory. What you do not get is two machines working the same second.
• What you do get in parallel is a pipeline: one machine rendering the next chunk while the other decodes and upscales the last one. The VAE decode is a third of the wall on long clips, so that is real.
• And the plain win: on an RTX 5090 a 321-frame clip at 384×256 did not render at all before 0.5.95 — the bidirectional path asks for a single 2 GiB buffer binding and dies. Chunked, it is 354 s. Your 10-frame prototyping budget on the 24 GB machine is exactly the constraint this lifts.

Chunk length matters more than context: 13 latent frames holds, 5 drifts at every boundary no matter how much context you give it (measured — quadrupling the context cost 2.3× the denoise and changed nothing about the drift).

One caveat so you do not download 0.5.96 expecting magic: its measured 10% is on the wgpu/Vulkan path — your Macs run the Metal one and see none of it. The Mac-side equivalent of that fix is a separate piece of work, and now that I know what to look for (host preparation per call, not kernels) it is a much better-aimed one.

not sure how this fits in but its a great idea

https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod

(from page)

In MiniMax H3 you can give the AI a reference — an image, a video, even a GIF — to tell it "look like this". That's powerful, but every reference gets loaded and processed every time you generate, which is slow and can "bleed" its look into everything else in your video.

This pack lets you save that reference once as a tiny .safetensors file (a "mod"), and then reuse it as many times as you want, whenever you want:

Save once — take your image/video/GIF, hit Extract, and it becomes a small file on disk. No need to keep the original clip around or load it again.
Reuse anytime — load the mod in one node, like picking a LoRA. Adjust how strong it is with a simple number (strength), or blend a few mods together (face + style + outfit, etc.).
No more heavy reference loading — you can leave the H3 reference input empty and inject the mod through the conditioning instead. Faster generation, and the reference only affects what you want it to affect.
No training needed — this is not a LoRA you train for hours; you just encode your reference and save it.

https://huggingface.co/iamkaikai/MiniMax-H3-Single-Frame-VAE-500K

this one is NUTS. lets you export a single frame . would be excellent for so many use cases

https://github.com/T8mars/comfyui-minimax-h3-blockcache-T8

An experimental custom node "MiniMax H3 Block Cache (T8)" has been released to accelerate the processing of ComfyUI's video and audio generation model "MiniMax H3".

▼Key Points
・A caching mechanism that skips processing of specific blocks during stable audio and video to achieve speedup
・If the difference between audio and video exceeds the threshold, it automatically performs full computation for all layers
・You can choose CPU, which saves VRAM, or GPU, which reduces transfers, as the cache destination
・Lossless reproduction with the same seed is not guaranteed, and quality and speed may vary depending on settings and environment

▼Use Cases
・When generating videos with MiniMax H3 and you want to improve generation speed while tolerating slight variations in quality.

Triage of the three new finds, honestly, and the two-Mac question you asked that I never answered.

Two machines on one prompt — here is what is real. Gemini is right about the narrow case and wrong about the useful one. Splitting one clip's DiT across a LAN does not pay: every denoise step would push the full latent hidden across the wire, hundreds of MB per step, and the link becomes the model. But two other splits work with what ships today:

• Batch throughput: two machines rendering alternate clips is 2× today, no new code — same file, same command, different prompts.
• Pipeline over RAVEN chunks: the streaming container generates causally, chunk by chunk. That makes a real two-machine pipeline possible — the 24 GB M4 denoises chunk N while the M1/64 decodes chunk N−1 through the VAE, so the big VAE never blocks the DiT. That one needs plumbing (a chunk handoff), and it is the first split I would build, because the pieces already exist: the LLM side of the engine has had a network worker (cortiq worker + --peer) since the phone-split work, and the chunk boundary is a few MB, not hundreds.

Single-Frame-VAE-500K — the right integration is a --vae override on animate, so any VAE fine-tune drops in beside the packed one. Queued; the pack path already carries the VAE separately (animate-pack --music-vae proves the seam exists). Until then it can be packed into its own container the same way RAVEN was.

blockcache-T8 — a training-free speedup that skips blocks whose activations barely moved between steps. Worth real attention here, because on this engine the denoise is ~80% of a render, so block-level caching attacks the dominant cost. It is also the kind of change that MUST ship with an A/B gate (the plane-cache experiment we ran earlier measured as pure noise, and only interleaved runs showed it) — so it goes on the list with a measurement harness, not as a quick port.

M3Mod (reference isolation) — two separate ideas inside it: caching the encoded reference (cheap here — the vision block encode is a small slice of a render, but in serve mode reuse across requests would be free and real), and bleed isolation, masking the reference tokens out of chosen blocks. The second is the interesting one and is honest new work on the attention mask, not a loader feature.

No dates promised on any of these — but this is the order they earn: chunk pipeline, --vae override, blockcache with a gate, ref isolation.

awesome! chunk pipeline would be the most impactful. in theory wouldnt this allow for longer frame renders with shared context?

like mac mini 24gb kicks off and takes a 100 frame prompt (impossible now without referring the owner to swap abuse hotline), but this is shared by the m1/64 gb on the network. and they split the load. i understand that you saying this is possible.

but im wondering this would theoretically allow for endless rendering with SHARED initial prompt context. like 'make me a waterfall scene that lasts half hour with xyz specific attributes and conditions' and it maintains consistency as it bounces the chunks between machines until its done. I wonder if this could work like this?

Mostly yes — and the part of your theory that is right is the important part, so let me draw the line between what falls out for free and what does not.

The RAM ceiling stops limiting LENGTH. That is the real unlock. Today a render holds every frame's latents at once, which is why 100 frames on a 24 GB mini means the swap-abuse hotline. The streaming container generates causally, chunk by chunk (13 frames a chunk — 13 exactly; 5 collapses, we measured), and a chunk only conditions on the tail of the previous one. Peak memory is one chunk plus its conditioning window, regardless of total length. So a 100-frame render on the mini becomes a time cost, not a memory cost — before any second machine enters the picture.

What "shared context" actually is. Two things carry across chunks: the prompt conditions every chunk (your waterfall stays a waterfall with its xyz attributes for the whole half hour), and the conditioning tail makes motion continuous across the boundary. What does NOT exist is global memory — a kayaker who paddled through at minute 2 is not remembered at minute 20 unless the prompt or a reference pins him. Consistency = global style + local continuity, not a growing world model.

The enemy at half-hour scale is drift, not memory. Each chunk extrapolates from generated (imperfect) frames, so error compounds — exposure, palette and texture wander over many chunks. Minutes are plausible; half an hour untested and I would expect visible drift without re-anchoring. The honest fix is periodic anchors: re-inject the first frame (or a reference sheet) into the conditioning every N chunks, which the ref machinery can already express. That is the piece I would build with the chunk handoff, not after it.

What two machines buy — and what they cannot. The chunks are causal: chunk N needs chunk N−1's latents, so the denoise itself is serial and two machines cannot split it. What the second machine takes is everything else: VAE decode of chunk N−1 (the 5 B-parameter half of the walltime), audio, muxing — running while the mini already denoises chunk N. That is a real 1.5×-ish on wall clock and, more importantly, it keeps the mini's 24 GB free of the VAE entirely, which raises its chunk budget. Not 2× on denoise — physics says no — but endless-length at the mini's denoise speed with the M1/64 hiding the decode: yes, that is the design.

So your half-hour waterfall decomposes into: streaming container (exists) + chunk handoff over the network worker (queued, first in line) + periodic re-anchoring against drift (queued with it). No promised dates, but nothing on that list is research — it is plumbing on parts that already run.

you NEED to read this! https://nvlabs.github.io/Sana/Sol-Engine/H3-Super-Acceleration/

the loras and such are posted at the bottom of the paper. can we implement this???

Read it. Good find — and worth taking apart before anyone gets excited about the number, because you are already holding more of this than the headline suggests.

What the pipeline actually is. Stage 1: MiniMax H3 with the lightx2v 4-step Turbo LoRA, drafting at 896×512. Stage 2: upsample, then a 3-step LTX-2.5 refinement pass to restore detail at the target resolution. The 22.2×/27.7× headline compares against a ~50-step SGLang baseline on a GB200.

What you already run. Stage 1 is this repo. MiniMax-H3-Turbo-cmf ships with that same Turbo LoRA fused — the 4-step draft is your everyday render, which means most of the 22× was in your hands before NVIDIA wrote the page. And our current --upscale (0.5.93) is the cheap cousin of their stage 2: draft low, learned latent upsample, decode once.

What is genuinely new and implementable. The refine trick: instead of decoding the upsampled draft directly, re-noise it lightly and run ~3 denoise steps through LTX-2.5 at the target resolution. Every part of that exists in this engine — H3 and LTX both run natively in the one binary (nobody else has that; NVIDIA's own pipeline straddles two frameworks), and LTX's --two-stage already does re-noise-and-refine for its own drafts. The new work is the cross-model handoff: H3 frames → LTX's VAE space → partial-strength denoise. Plumbing on parts that run, same class as the chunk pipeline — and the two compose: a refine stage can run per chunk, on the second machine, which is your endless-waterfall design acquiring a quality stage for free.

What cannot be implemented from this page. Sol-Attn. It is named, timed (0.985 s of the refiner), and not released — no code, no weights, no description. The recipe works without it (it is their stage-2 attention speedup, not a correctness piece); our LTX runs its own attention. If NVIDIA publishes it, we will read it then.

The small gem at the bottom. taehv — tiny approximate VAE decoders for H3-class latents, a few MB, open. Ported, that gives instant preview decode while the real VAE runs elsewhere — and it slots straight into the --vae override already queued from your Single-Frame-VAE find.

Honest expectation. Against what you run today (turbo draft + latent upscale), the LTX-refine stage is maybe a 'sharper output at similar cost' or 'same output faster at high res' trade — call it 1.5-3× at 768p+, not 22×. The 22× is against a baseline neither of us has used in months.

Queue, updated: chunk pipeline → LTX-refine handoff (this) → taehv preview → blockcache-with-a-gate. No dates promised, but the first two share most of their plumbing.

you are the boss man!

1- "tiny approximate VAE decoders for H3-class latents, a few MB, open. Ported, that gives instant preview decode while the real VAE runs elsewhere " - YES this would be impactful, having a peek at the direction the diffusion is going will save tons of times than waiting for output. right now the work around for this 'draft' mode is decreasing the size and frames both of which are not really representative - esp since prompt need more frames to complete the instruction (ie 'girl gets up out of bed and puts on jacket') - you cant express than in 5 frames. having a preview at low quality without waiting for an entire sequence to render would help

2- "The enemy at half-hour scale is drift, not memory. " - as usual "its all about the harness" . yes prompting and ref images between chunks can enhance its memory. quite easy to automate this too. just use another model to read/see the output made (esp if we can get some low rez inprocess frames for it) it can update its 'facts' about the scene. if it sees the kayaker coming acorss, the next prompt for the chunk can ad that to its 'world understanding'. in fact maybe something as simple as 'offscreen do not show: - kayaker, mermaid, etc etc. if the prompt is 'you are a drone filming a waterfall in smooth different angles, it shoudl then pick up the kayaker as it comes into frame. in sort the 'world knowledge' can easily be fixed with 'machinery' as you call it

throw a 'buy me a coffee link' and ill buy you a coffee BOSS MAN :))

The NVIDIA two-stage recipe runs. Shipped in 0.5.100 (one flag was missing — the handoff), verified end to end on a 5090, and the A/B is in this repo for your own eyes:

• samples/refine_ab_draft_512x288.mp4 — the 4-step Turbo draft, 40 s
• samples/refine_ab_refined_1024x576.mp4 — after the LTX refine

The recipe

cargo install cortiq-cli   # 0.5.100+

# stage 1: the Turbo draft, frames dumped for the handoff
cortiq animate mmh3-turbo-clipproj4b-fl2va-v2-q4tp.cmf \
  --prompt "a woman in a red raincoat walks down a neon-lit street at night, rain, reflections, cinematic" \
  --width 512 --height 288 --frames 39 --frames-dir draft/

# upscale the frames 2x with anything bicubic (PIL, ffmpeg, sips)
# and trim to 33 — the two temporal grids differ (H3 renders 17k+5,
# the LTX VAE takes 8k+1), so 39 draft frames become 33 refined ones

# stage 2: LTX re-noises the draft and denoises it at full resolution
cortiq ltx-video --model ltx25-q4tp.cmf \
  --prompt "…same prompt…" \
  --video draft2x/ --video-strength 0.45 \
  --width 1024 --height 576 --frames 33 --out refined.y4m

--video-strength is the dial: 0.45 keeps the draft's composition and motion and lets LTX reinvent the detail; higher repaints more, 0.2 barely touches it.

Numbers, 5090, honest

That last line is the finding that matters more than the recipe: 76% of the refine is the two VAEs (encoding your draft in, decoding the result out), not diffusion. Sol-Attn would have attacked the 20%. The 76% is exactly what the two-machine chunk pipeline removes — your mini denoises, the M1/64 owns the VAEs — so this measurement moved that design from "sensible" to "measured".

Practical notes for a 24 GB Mac: the two containers (14.5 + 22.1 GB) don't want RAM at the same time — run the stages sequentially (draft, free, refine), which the two-command shape already is. And keep the same seed and prompt across both stages; the refine leans on the text as much as on the frames.

Grid resampling in the handoff (so you never count frames yourself) and the taehv preview decoders are next in the queue.

https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union

This one would be incredible to have. easy way to control the videos

tweet with example video here

https://x.com/aisearchio/status/2092032788577071565