A 4K Video Matte Used to Want 80 GB of VRAM. Pruning Brings It Under 4.

Every pipeline that pulls a character out of generated footage runs into the same compromise. The clip arrives at 4K, the matting model will not fit at 4K, so you downscale to 1080p, matte there, then scale the alpha back up to fit the plate. Hair turns into a soft grey fringe. A strand of it drags a thin line of the old background across the new one. On a bright background the seam is the first thing anyone notices.

The reason has always been memory, not ambition. MatAnyone 2, the previous version of the same method, needed an estimated 79.69 GB of GPU memory to matte at 3840×2160 and ran at roughly 1.1 frames per second. That is a datacenter problem, not a desktop one.

A paper from Shuyun Wang, Shangchen Zhou, Peiqing Yang, Yuekun Dai, Xin Yu and Chen Change Loy (arXiv 2610.35724) attacks that number directly. Their extension, MatAnyone-E, reaches native 4K with peak memory of 6.08 GB on an A800 without compilation, and 3.89 GB with torch.compile. That is 13.1× less memory than MatAnyone 2 at the same resolution, at 7.4× the speed. On a consumer RTX 3090 with compilation it holds 4K at 9.0 fps and 3.16 GB.

The savings are more interesting than the total, because the same idea applies well beyond matting.

Why matting at 4K was so expensive

Memory-based video matting works by comparison. The model keeps features from earlier frames, then matches the current frame against that stored memory to keep the matte stable across a shot. That is what stops the alpha from crawling on the edges between frames.

At 1024×1024 the number of pixel-to-memory comparisons is manageable. A 4K frame has about eight times as many pixels, and each one can look at many stored frames. Cost climbs faster than resolution, which is part of why the memory figure jumps from 1.60 GB at 1024 to 79.69 GB at 4K.

The insight in the paper is that almost none of that work changes the result. Most pixels in a frame are easy: flat background, flat foreground, no ambiguity. Only the hard ones need full resolution, and those are concentrated on hair, fur, glass and motion-blurred edges. A uniform pass spends the same compute on a wall as on a strand of hair.

What the sparse version does instead

MatAnyone-E keeps the existing architecture and replaces the two stages whose cost grows fastest with resolution.

It runs a cheap low-resolution propagation pass first. That pass is not the final answer, but it is good enough to expose where the model is unsure. Uncertainty selects which query pixels deserve a detailed look, which the authors call query pruning. The same pass produces affinities that identify which stored memory frames are actually relevant, which is memory pruning. Full-resolution matching then runs only on those small sets of query and memory pairs, and everywhere else the upsampled low-resolution readout is reused.

The published gate numbers are aggressive. 5.9% of queries pass the uncertainty threshold. 15% of the memory is retained for the refined read. Multiply those together and 99.1% of full-resolution attention is skipped.

The second change is a lightweight decoder, designed around measured runtime bottlenecks and trained to reproduce the representations of the original decoder through feature distillation. Decoding a 4K alpha channel was a real cost on its own, and this removes most of it.

The numbers, including the ones that matter less

Throughput and peak memory, compiled where noted, from the project page:

• 1024×1024: 47.3 fps at 0.61 GB, against 29.5 fps at 1.60 GB for MatAnyone 2
• 1920×1080: 37.6 fps at 1.07 GB, against 13.6 fps at 5.47 GB
• 2560×1440: 25.1 fps at 2.02 GB, against 5.6 fps at 16.20 GB
• 3840×2160: 13.6 fps at 3.89 GB, against an estimated 1.1 fps at 79.69 GB

On consumer hardware, an RTX 3090 with compilation gets 29.4 fps at 1080p using 0.87 GB, and 9.0 fps at 4K using 3.16 GB. An H200 NVL reaches 73.1 fps at 1080p and 28.6 fps at 4K.

Two of those figures are estimates. The project page marks MatAnyone 2's 4K memory and frame rate as extrapolated, so treat the 80 GB headline as an order of magnitude rather than a measurement. The 3090 numbers are also compiled inference, and compilation changes what fits.

Quality is the part that decides whether any of this is usable. Mean absolute difference against ground truth, in units of 10³ where lower is better:

• VideoMatte HR: 4.14 for MatAnyone-E, 4.10 for MatAnyone 2
• YouTubeMatte HR: 1.62 against 1.61
• CRGNN: 4.28 against 4.24

That is a rounding difference. The speedup did not buy its headroom by giving up edges.

Put in production terms, a 30-second shot at 24 fps is 720 frames. On a 3090 that is about 80 seconds of matting at native 4K, or 24 seconds at 1080p. At those speeds the matte stops being the step you schedule a day around.

Why this lands differently for generated video

Traditional rotoscoping exists to separate an actor from a plate that was shot once. AI video has a different problem: the character was generated with the background, and the two are entangled. If you want that character in a second shot, on a different set, under different light, you either re-generate and hope the identity holds, or you extract the character and composite.

Extraction is the more controllable path, and it lives or dies on the matte. Hair, glasses, a translucent sleeve, smoke drifting past a shoulder: these are where an AI composite announces itself. A matte that only holds at 1080p forces you to either deliver at 1080p or accept a soft edge on a 4K master.

There is a second, less obvious use. Re-rolling part of a shot becomes cheap when you can isolate the moving subject from the plate. Fix the character, keep the background, and the shot does not need to be regenerated end to end. That workflow needs alpha at the resolution of the plate, not a proxy.

The transferable part

Strip the matting specifics and the architecture is a budgeting rule: run a coarse pass to find out where detail matters, then spend compute only there. That is the same shape as a proxy edit, and the same shape as HiRes Fix in stills, where a cheap first render decides where a second pass is worth running. Tiled upscalers that only re-render uncertain tiles are the same idea again.

It is also a warning about how we read hardware budgets. A 79.69 GB figure tells you the method is wrong for a desktop, not that high-resolution matting is a datacenter feature. The same correction is available in local generative fill, where the ceiling came from model tier choice rather than from the GPU.

What to actually change in your pipeline

Match the matte resolution to the delivery. If the master is 1080p, matte at 1080p and stop paying for pixels nobody sees. If the master is 4K, the old downscale-and-upscale trick is now a choice rather than a requirement.

Inspect at 100% on the three places mattes fail: fine hair against a bright plate, the edge of anything transparent, and any frame where the subject moves fast enough to blur. A matte that looks clean in a fit-to-window preview is not information.

Keep the alpha linear until the last possible step, and check premultiplication before you hand the comp to an editor. Most "bad matte" complaints turn out to be a gamma problem on the edge pixels.

Do not matte after a denoise pass. Denoising destroys the low-contrast detail at soft edges that the matting model depends on, and you get a cleaner-looking result that is worse.

This is a research release, not a ComfyUI node you can install this afternoon. The numbers are worth knowing now anyway, because they set expectations for what the tooling will look like when it does land: compositing stops being the step that forces you to work below the resolution you generated at.