PiD Puts Upscaling Inside the Decoder, and That Moves Where Detail Comes From
NVIDIA replaced the VAE decoder with a four-step pixel diffusion model that turns a 512px latent into a 2048px image in one pass, at 13 GB peak memory. The useful part is not the speed claim. It is that a caption now steers the pixels you get, and that the wrong VAE produces a channel-mismatch crash instead of a warning.
AI-assisted draft. Reviewed and edited by the Phosphene team before publication.
On this page

Every latent image pipeline ends with the same anticlimax. The sampler finishes, and then a fairly small autoencoder has to turn a compressed representation back into something you can look at. That decoder was trained to invert the encoder, not to invent anything, so at high resolution it is the weakest link dressed as plumbing.
PiD is NVIDIA's attempt to delete that step as a separate stage. The name is a contraction of Pixel diffusion Decoder, and the pitch is narrow enough to check: replace the VAE decoder with a conditional pixel diffusion model, and you get decoding and 4x upscaling in a single generative pass.
The Japanese daily roundup where this surfaced framed it as a generic 4x upscaler that works on any model. That undersells the interesting half and oversells the easy half, so here is the version with the caveats attached.
What it actually changes
Three mechanisms do the work.
The first is the obvious one. Instead of a convolutional decoder, PiD denoises directly in high-resolution pixel space, conditioned on the latent. Because it is generating pixels rather than inverting an encoder, it can synthesise detail that was never in the latent instead of interpolating what is there.
The second is the reason it fits into a pipeline rather than bolting onto the end of one. A lightweight sigma-aware adapter injects noise-corrupted latents into the pixel backbone. That means PiD can decode a latent that is not fully denoised, so the base model can stop early and let the decoder finish. The project demonstrates decoding at intermediate sampling steps, not only at the final one.
The third is distillation. The model is DMD2-distilled down to four steps, which is what makes a second diffusion model in the graph affordable in the first place.
The numbers, and who produced them
On the project's own 512 to 2048 comparison, end-to-end decoding latency:
- Real-ESRGAN: 62 ms
- PiD: 211 ms
- LUA: 369 ms
- TSD-SR: 725 ms
- InvSR-1: 1,018 ms
- SeedVR2: 1,238 ms
Peak memory for that decode is 13 GB on a consumer RTX 5090, and 210 ms on a GB200. In a pairwise judge comparison run with Gemini-3-Flash, PiD was preferred over the other five between 85 and 99 percent of the time, depending on which baseline it was up against.
Read that table honestly and the first row is the one that stings. Real-ESRGAN is more than three times faster than PiD and will still be the right call for a contact sheet. PiD's win is against the diffusion-based super-resolution stacks, where it is up to 5.9 times faster than SeedVR2 while scoring better. It is not a free lunch against a GAN, it is a different quality tier at a compute cost that stops being silly.
These are the authors' measurements on the authors' comparison page. Nobody independent has published a bake-off yet, and the visual examples are curated by the people who trained the model. Treat the direction as established and the exact margins as provisional.
Why the caption input is the real story
In the ComfyUI node set, the decode signature is latent + caption + sigma -> image. That caption is not decoration. The decoder is text-conditioned, which means the words you pass it influence the pixels it regenerates.
That is a genuinely new control surface, and it cuts both ways. On the good side, an upscale of a 512px render can be told what it is looking at, and small faces and fine structures come back better described than guessed. The v1.5 checkpoints were trained with more anime data and more small-face data, which is the kind of change that shows up exactly where upscalers usually fail.
On the bad side, a caption that contradicts the image gives the decoder licence to draw the contradiction. Upscale a photo of a blue jacket with a caption that says red and you have handed a generative model a reason to change your product shot. If you are used to thinking of upscaling as a safe operation, this is the part to unlearn. The caption belongs with the prompt and the seed in your reproducibility record, in the same way we argued for keeping the full settings alongside the image.
The VAE trap that produces a crash instead of a warning
This is the most practically useful thing I found, because it is the mistake you make on the first attempt.
PiD decodes into pixels. There is no traditional VAE at the end of it, and the node that would normally sit there has to be told so. In ComfyUI issue 14154, a user decoding a FLUX.2 Klein latent hit this:
RuntimeError: The size of tensor a (3) must match the size of tensor b (128) at non-singleton dimension 1
Those two numbers are the mismatched channel counts: three channels of pixels against a 128-channel FLUX.2 latent, at the point where the VAE applies its scale and shift. The answer from the ComfyUI side is blunt: decode with the VAE that comes with PiD, and using FLUX's VAE here is an error. A community explanation puts it more usefully still: set the VAE loader to pixel space, because the point is that PixelDiT is doing the decoding job itself.
For the tiled upscale node the requirement flips back the other way. That node cuts the image into tiles, encodes each tile with the matching VAE, runs PiD, blends, then resizes. So it does need a real VAE file per backbone family: ae.safetensors for Flux1 and Z-Image, the FLUX.2 autoencoder for the FLUX.2 family, and so on. Two different requirements in the same node pack, which is exactly the sort of thing that eats an afternoon.
Sizing, in the node's own terms
Released checkpoints use a native 4x scale, and the profile you pick decides the input class:
2k: trained at 2048px. A 512-class base becomes 2048, so 512x512 goes to 2048x2048.2kto4k: a 1024-class base becomes 4096.2kto4kwithversionset tov1.5: the same range, with better colour accuracy, no corner grid artifacts, and more anime and small-face training. Less sharp than2kat 2048.
Recommended sampling settings in the node are four steps, CFG 1.0, and a scale of 0 or 4. Version 1.5 wants ComfyUI 0.28.0 or newer, so check your build before downloading a checkpoint it cannot load. Precision is constrained too: v1 has fp8 only for parts of the FLUX families, and v1.5 offers bf16 or int8, with unsupported combinations raising an error rather than falling back. That last behaviour is the correct one, and worth naming because so much of this ecosystem prefers a quiet downgrade.
Backbone coverage, if you are planning around it: FLUX, FLUX.2 and its Klein variants, Z-Image and Z-Image-Turbo, SD3, SDXL, and Qwen-Image. In the native Comfy packaging, SDXL and Qwen-Image are 2kto4k only. The DINOv2 and SigLIP latent spaces used by RAE-style models are not part of the native Comfy packaging.
The licence is the blocker, not the memory
The code is Apache 2.0. The model weights are a different document. The node's readme states that NVIDIA's weights ship under NSCLv1 and are limited to non-commercial research and evaluation use, and that sentence sits in its PixelDiT generation section rather than beside every checkpoint. The safe reading is to open the model card for the exact checkpoint you plan to run before it goes near client work.
If you generate for yourself, this is a footnote. If you generate for clients, it is the whole decision, and it belongs next to the other licence questions that decide whether a model can go into paid work, the way the P-H3-Image territory limits did for editing.
Where I would use it
- Finishing a 512-class render for print or a large hero. This is the clearest fit: one pass, 13 GB, and detail that is generated rather than smeared.
- Ending a sampler run early. If a base model converges by step 20 of 28, letting the decoder absorb the last eight steps is real time back, and the sigma adapter exists precisely to make that legitimate.
- Anime and small-face work, on the v1.5 checkpoints, with the caption written as a description of what is already in the frame.
- Not for previews. Real-ESRGAN at 62 ms wins that job and always will.
- Not without a caption discipline. Whatever you type there is now part of the render.
The uncomfortable question this raises is a good one. If the decoder is generative, then "upscaled" and "regenerated" are no longer cleanly different operations, and the honest label for a 4x pass is closer to a re-render that kept the composition. That is worth knowing before the client asks how the final file was made.
Sources
- 旧世代GPUでDeepSeekを高速動作させる最適化術と、ComfyUIのChatGPT公式統合で変わるAI制御 (Japanese original, daily AI roundup where PiD surfaced, Ai-Hakase on note)
- PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion (NVIDIA project page)
- nv-tlabs/PiD: code, checkpoints and inference entry points
- Merserk/ComfyUI-PiD: node set, backbone matrix and output size guide
- ComfyUI issue 14154: channel mismatch when decoding a FLUX.2 latent with the wrong VAE