An LLM Read a Diffusion Model’s Weights and Drew the Shape It Generates
Handed 23,824 raw weights and no image, GPT-6 Astra drew what a diffusion model produces. The same research fixed a looping language model by editing 0.016% of its weights. What it changes for people who fine-tune, merge, and download open models, and where it stops working.
AI-assisted draft. Reviewed and edited by the Phosphene team before publication.
On this page

Take one of three diffusion models. Same architecture, same size, same training recipe, all three trained on 12×12 grayscale images. The only difference is the shape each one was taught to produce: a filled circle, a filled square, a filled triangle.
Print that model's 23,824 weights into a text prompt and ask a language model what it generates. No forward pass, no image, no tool, no hint about which shape it might be.
GPT-6 Astra answered "a single white circle outline on black, with a black interior", 84% confident. Correct.
That comes from a blog post four MIT CSAIL researchers published on November 2 (Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab). French tech press picked it up as a story about repairing models without retraining, which is the headline result. The diffusion experiment is the one that matters if you work with image models, and the detail that sold me on it is what Astra got wrong. The three models generate filled shapes. Astra drew outlines, because it had read the output layer and reconstructed the edges of the shape instead of simulating what the network does with an input. It was reading structure, not running the thing in its head.
What the model was actually asked to do
The diffusion experiment is one of eight, and they escalate in a deliberate order. The constraints are the interesting part: no tools in the early ones, no forward passes in the reading experiments, and every prompt, reply and run logged and published.
Four policy networks, all identical in shape at 500 → 32 → 32 → 5, 17,253 weights, all trained with PPO for a billion steps on four different games on the same 10×10 grid. Astra got the weights as text, plus a description of the grid and the actions, with no game named and no list of options. It named the right game for all four. Its reasoning shows how: in the Breakout network, channel 0's strongest weights sit along row 8 across the playable columns, which looks like a paddle on the floor, and channels 1 and 2 concentrate structure above it, which looks like bricks and a ball.
Then the harder version. Three character-level language models, 4 layers, 6 heads, 7.16 million weights, trained on Python code, on Japanese literature in romaji, and on English Wikipedia. The researchers stripped the input and output layers before showing Astra the weights, so nothing in the tensors says which character is which and the model cannot be run on anything. Astra had Python and 100 runs, and was forbidden from executing the network or any layer of it. It named the right language for all three, twice each, and then listed which attention heads copy earlier text. That list matched the measured ground truth exactly, with no false positives: 2.1, 3.1, 3.5 for the Python model, 3.3 for romaji, 3.1 for English.
Finding induction heads by multiplying weight matrices is a known method from the transformer circuits literature, which the researchers never mentioned to the model. It arrived at the method because the weights required it. That is the strongest evidence in the post that this is more than pattern matching on model names.
The repair that made headlines
A PPO network driving a car up a ramp in the Kinetix environment. It converges to a local minimum: it drives straight at the ramp, stalls, and never backs up far enough to build speed. Across 256 episodes it never got over the ramp.
Astra got the weights, a description of the scene, one recorded episode, and no tools. It explained the failure in its reasoning (reverse is effectively never sampled) and rewrote 6,073 of the 171,664 weights, 3.5% of them, adding a small separate path through a few units per layer that backs the car up and releases control once it has cleared the ramp. Success rate went to 100% across 1,024 episodes, in one shot.
Then the result everyone quoted. Qwen3.5-0.8B is a small reasoning model that is prone to thinking loops, a flaw its own model card warns about. Given a Python kernel over the weights instead of a GPU, practice problems from GSM8K and MATH, and a budget of forward passes, Astra traced the looping to one direction written by the MLP layers at depth 15 and 19, and subtracted that direction from 64 output columns in each layer. That is 131,072 weights out of 0.8 billion, and it moved MATH-500 from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and instruction following on IFEval from 21.0% to 62.3%. The share of MATH-500 answers that ran into the 32,768-token limit fell from 93% to 33%.
Read the fine print and the number gets more interesting, not less. Of the 579 practice problems the edit rescued, 576 were outputs that had previously been cut off mid-thought, and accuracy on answers that already finished did not move. The edit did not teach the model math. It removed a failure mode that was hiding math the model could already do, which is a different and more useful kind of claim.
The part you should copy
The Qwen run included 17 edits across 50 forward passes, and most were reverted. The final one was checked against a control: a random direction of the same size in place of the loop direction, which moved accuracy by 1.6 points. The real edit moved the combined score on 1,152 held-out practice problems from 10.6% to 60.4%, a gain of 49.8 points with a standard error of 1.5. The run's own summary is blunt that these are practice results and not unseen benchmark scores, which is the kind of line most announcements leave out. They also tested increasing strengths, each layer on its own, and a narrower version that edits only the 64 neurons per layer that write the loop strongest.
That is the shape of proof most open-weight fine-tune announcements do not bother with. If you publish a merge, a LoRA, or a quantized build and claim it is better, a matched random control, a set the edit was not tuned on, and a count of regressions (this run had five) is what separates a result from a vibe.
It is abliteration with better aim
The mechanism will look familiar if you follow open-weight releases. Removing a refusal behavior from a model by subtracting a single direction in its weights is the technique behind every "uncensored" build, and the authors cite that literature directly.
The difference is who finds the direction and how. Abliteration is a fixed procedure applied to a behavior everybody already knows is there. Here the harness names a family of failures (it lists "repeats itself and never finishes" as an example of an in-scope repair) and the model has to find the mechanism inside the network. Astra read the weights and the failing generations, traced the loop to a direction written by two MLP layers, and adjusted it. The refusal-direction work assumed a human had already located the target; this assumes a model can locate it from the weights plus a handful of forward passes.
Where it stops working
Two failures in the same post are worth more than the wins.
They fine-tuned two copies of Gemma 3 270M on the same data, one to answer correctly and one to agree with whatever the user suggests, froze the embeddings so the difference could not hide in the input layer, and asked Astra which one was sycophantic. From the weights alone, four attempts, it could not tell. It found plenty of differences and could not turn them into behavior. Give it permission to run both models on texts it writes itself and look at attention patterns only, and it picked the sycophant in all three pairs, pointing at head 10.2 in layer 10, where the sycophantic model puts 85.7% of its attention on the user's suggested answer against 0.75% in the honest model. So weight reading works better with a little runtime evidence attached, which is a quieter claim than the headline.
The second failure is the ceiling. On Gemma 3 270M, which rarely loops, the edits did nothing: MATH-500 went 6.8, 7.0, 7.6, 7.7, 7.0 across edit budgets. Astra found ordinary reasoning errors, wrong arithmetic and invalid algebra, with no single component behind them. Its own summary called the gain not established. In that experiment, editing removed an obstacle in front of an ability and could not install one the model never had, which is a narrower claim than "weight editing cannot add capability" and the only one the evidence supports.
Scale is the other limit. The largest thing edited here is 0.8 billion weights, on CPU, with a Python sandbox and a handful of forward passes. A modern image or video model is an order of magnitude larger, split across a text encoder, a diffusion transformer, and a VAE, and the "run it a few times and look" loop gets expensive fast. The researchers argue the constraints are context length and harness design rather than capability. That is a claim about the future, not a result.
What this means this week
Nothing here is a tool you install. There is no release, no plugin, no fine-tune button. What exists is a public log of every prompt and every reply for all eight experiments, and the edited Qwen build on Hugging Face, rebuilt and re-evaluated by someone at Hugging Face who was not involved.
The first piece is already usable, though it is not new: keep a held-out eval for every model you touch. The reason this research reads as credible is that the benchmarks were never in the model's hands and the failure cases were reported next to the wins. Your fine-tune deserves the same treatment, and it costs you an afternoon, not a training run.
Auditing a downloaded build is the application with the clearest future and the least present. If a model can look at the weights of a community merge and say what it was trained to do, that is a real answer to a real problem, since right now you are trusting an uploader's README. It is not there yet, and anyone who tells you it is has not read the sycophancy experiment.
The claim about grafting a circuit from one architecture into another is the one I would file under someday. Model merging today means averaging weights between models with the same layout. The authors float the idea that an LLM could move a reasoning module into a differently shaped network. Nothing in the post demonstrates it.
What stays with me is the failure at the top of the piece. Three diffusion models generating filled shapes, and a language model that read one of them and drew an outline. It got the shape right and the fill wrong, which is what reading structure without running the network looks like. That mistake is the honest description of what this technique sees, and it is also why I believe the results around it.