Your Fast Attention Path Was Silently Falling Back 384 Times per Clip

A Japanese AI news digest flagged an optimization write-up last week with a number that reads like a typo: LTX-2.5 generating a ten-second 1080p clip with audio in 43 seconds instead of 130. The digest summarized it as a speed story. The original write-up is more useful than that, because two of its three findings are not about speed at all. They are about silent failure.

The setup: one RTX PRO 6000 Blackwell, same model, same image and prompt, same 600 W power cap, warm runs, board power sampled every tenth of a second. Stock ComfyUI: 130.1 seconds and 19.2 Wh per clip. Rebuilt: 42.9 seconds and 6.9 Wh. Roughly three times faster, and less than half the energy for the same output.

Here is what was actually wrong, in the order the write-up presents it, which is also the order of how badly each one generalizes.

Finding one: the fast path declined to run

The first thing the profiling found is not a slow kernel. It is a fast kernel that never executed.

When you use a guide frame, the pipeline adds an attention mask for the first pass. SageAttention, the fast attention backend, falls back to the slow path whenever it sees a mask. In this pipeline that produced 384 slow attention calls per clip. The optimization was installed, enabled, and correctly configured, and it ran zero times.

The fix is a math argument rather than a speed hack: the mask only re-weights one frame's worth of keys, so run the fast path over everything, do one small exact pass over just those keys, and merge the two results. Half the attention time, identical output.

Sit with how ordinary that failure is. Nobody forgot to enable SageAttention. The mask is a feature of the guide-frame workflow, and the fallback is a defensive default in the backend. Two reasonable decisions, composed, quietly deleted a 2x speedup. You would never find it by reading settings, because no setting changed.

The practical takeaway: a backend being selected is not the same as a backend being used. If your graph branches on input shape, precision, mask presence, or resolution, at least one path is slower than you think. Instrument the call counter once. It is the cheapest profiling you will ever do.

Finding two: decode and save was 38% of the clip

This is the finding I would hand to anyone optimizing a generation pipeline, regardless of model.

Decoding and writing the output took 49.7 seconds of the stock 130.1. Two separate causes, both structural.

Tiled VAE decode overlaps tiles to hide seams, so it does nearly three times the arithmetic the frame needs. And 43% of the decoder's time was not mathematics at all, it was memory traffic: a run of small kernels, each one writing roughly a 17 GB tensor for the next kernel to read straight back.

The rebuild went whole-frame decode, fused CUDA kernels, a rebuilt decoder attention, and frames piped directly to NVENC. 49.7 seconds became 6.6. It also landed closer to an fp32 reference than the stock decoder, because rounding once at the end beats rounding at every intermediate step.

Two things to steal here. First, measure your tail. People profile the transformer because that is the famous hard part, then discover that a third of the wall clock is post-processing. Second, tile overlap is not free seam-hiding, it is a 3x multiplier on decoder work; if your card has the memory for a whole-frame pass, the trade is usually worth it.

The caveat matters, though. Whole-frame decode needs the memory. This ran on a 96 GB card holding around 80 GB during a run. On a 24 GB consumer GPU the same change is not available, and a 3x improvement should not be read as a promise. This connects to a hard ceiling we covered in the Klein 4K pricing breakdown: past a point, quality per unit of compute is a memory-capacity problem, not a kernel-quality problem.

Finding three: the transformer had waste with a measurable address

The main model responded to the standard stack first. Official INT8 weights plus SageAttention nearly halved its time on their own. Then SageAttention 3 at FP4, and removal of two specific wastes: eighteen full-size copies of the same values per layer, and a 4.8 GB copy executed on every attention call.

Finally, block-sparse attention in 41 of the 48 layers, with the 7 layers where sparsity visibly hurt left dense, along with the frames that make loops seamless. That last clause is the one I would underline. Sparse attention changes what the model can see, and seam frames are exactly where it hurts.

The 18 copies and the 4.8 GB per-call buffer are the kind of thing no amount of reading the model card will surface. They only appear when you profile at the allocation level, which is why "just quantize it" plateaus.

The method is the most transferable part

The write-up states its own honesty rule, and it is stricter than most benchmark posts:

• Every change claimed as lossless had to produce byte-identical output to the original.
• Every change that alters the look, FP4 and sparse attention included, got a side-by-side and a human eye before it shipped.

That split is worth copying as a habit. "It looks the same" and "it is the same" are different claims, and mixing them is how optimization work turns into an argument about quality with no evidence on either side. Byte-identical is a hard gate you can automate. Everything else is a judgment call that needs a comparison image and a person.

How much should you trust the number

Four caveats, all from the write-up itself.

The hardware is a flagship 96 GB card, and the biggest single win depends on having enough VRAM to decode whole frames. On a mid-range GPU expect a different number and possibly a different bottleneck. The cloud comparison in the accompanying video is one run each, submitted through a hosted service, with queue time folded in. And the work was done by AI, with a person directing and a second model reviewing designs, which does not make the measurements wrong but does mean the code deserves a read before you run it.

That last point is not a dig. A 3x inference win produced by an agent on a graph nobody had profiled is a decent argument that profiling is exactly the kind of tedious, mechanical, high-leverage task agents are good at. It is also an argument for keeping the byte-identical gate, because the agent will not notice on its own that the picture changed.

What I would do in my own setup this week

Three measurements, none of which require a 96 GB card:

1. Warm up, then time the clip with a fixed power cap. Cold-start numbers are marketing.
2. Count fast-path invocations instead of trusting the node name. If the counter is zero, you found your 384.
3. Time the decode and encode stage separately from the sampling loop. If it is more than a quarter of the total, that is your next project, and it is probably the most boring one available.

If you are on a smaller card and this all sounds like someone else's problem, the MiniMax H3 speed notes cover the levers that actually move on 24 GB of VRAM, and the unified-memory workstation piece covers the hardware case where whole-frame decode stops being a luxury.