When AI debugs AI: repair or bypass?
I gave a frontier model the raw weights of a small model and asked it to fix a bug by hand. It did. Then I looked at what it had changed.
MIT CSAIL published a report this month called "Sparks of Weight-Space Programming". The claim: hand a frontier model the raw weights of a small model, a Python kernel and a budget of forward passes, and it will find why the small model misbehaves and fix it by editing the weights directly. No training. Their editor, GPT-6 Astra, changed 0.016% of Qwen3.5-0.8B and took its math score from 8% to 48%.
I wanted to see the method work on something it had not been tuned for. So I rebuilt the setup, pointed it at a different bug, used a different editor, and kept every hypothesis the editor wrote down.
The bug
Qwen3.5-0.8B writes SQL badly, but that is not the bug. The bug is what happens when you let it think first. On BIRD Mini-Dev, a text-to-SQL benchmark, 91% of its answers never arrive. The model starts a tidy "Thinking Process", lists the tables, and then re-reads the schema until it runs out of room at 8,192 tokens. This is question 806, about superheroes. The model restarts the same list 626 times.
Score with thinking on: 0.7% correct. This is the same kind of bug the MIT report fixed on math, so it is a fair test. Their own math edit, applied here, barely moves it.
The deal
The editor is Claude Fable 5.1, with no tools, no memory and no internet. It gets what the report gave Astra: a sandboxed Python kernel over the weights, 300 code runs, and 25 forward passes. A forward pass either answers practice questions (graded by running the SQL, original model against edited, side by side) or runs the model on a fixed text and hands back activations. It never sees the benchmark it is judged on. Then it is left alone for 9.8 hours.
Three attempts
- failed
Cut the copy heads. The editor generated 128 answers, confirmed the loop, and recorded attention over two loop transcripts. It found the heads that look one repetition back and copy, with layer 15 head 7 doing most of the work. It zeroed all four.
Layer 15 head 7 is the exact head the MIT run zeroed for math. Here it changed nothing. Still 100% cut off.
- failed
Silence the digit neurons. At each blank line the model chooses between "another indented bullet" and "next section number". Seven neurons in the last layer push the digit down by about 3 nats, one neuron per digit. The editor zeroed them.
A real, specific mechanism. Still 100% cut off. The editor wrote: "The looping is distributed and not fixable at one site."
- worked
Ask a different question. Not "why does the loop continue" but "why does the model enter this mode at all". At the very first thinking token, the closing tag
</think>is already a plausible next word. The editor found four neurons, one each in layers 17 to 20, that fire only at that position. It added to each a scaled copy of the direction the output layer reads as</think>.Four scalars. 4,083 weights. On seven fresh sets of 64 practice questions: 1.1% correct before, 28% after.
What it changed
Every MLP neuron writes one fixed direction into the model's running state when it fires. So you can place each neuron by that direction and watch which ones move.
Each dot is one neuron (300 per layer). Height is the layer. Sideways is alignment with the direction Fable added. Pink: Fable's four. Blue: the MIT math edit. Turn it yourself.
The four edits are one vector. Between any two of them the cosine is 1.0. Fable wrote the same push, toward </think>, into four different neurons, and they leave the cloud: from a cosine near zero to 0.44 to 0.69, where 98% of the model's neurons sit between -0.07 and 0.07.
The MIT math edit, in blue, changes 128 neurons and they do not move on this axis at all. They turn, but somewhere else.
That difference is the whole story.
Repair or bypass
There are two ways to make a loop go away. Fix the reasoning so that it ends: a repair. Or make the model write </think> at once and skip reasoning: a bypass. The model's chat template already has a flag for the second one. One count tells them apart: how much the model writes before the closing tag.
Fable's edit
Closes thinking on 150 of 150 held-out questions. Reasoning before the tag: 0 characters, every time.
BIRD, thinking on: 6.0% correct
Template flag, no edit: 7.3% correct
MIT's math edit
Closes thinking on 22 of 150. Where it works, the model reasons first: a median of 5,768 characters.
On math, beats the flag: 50.2% vs 45.8%
A repair, by this count
So the edit did not teach the model to stop looping. It taught the model to stop thinking, and it scores a point below the flag that does the same thing. A second Fable session, with no shared notes, reached the same place through different neurons (5.3% against 6.0%). Gradient descent, allowed to touch only the same 4,083 weights, lands there too, by a direction nearly orthogonal to Fable's. Same four neurons moved to random locations: nothing happens. The pick was real. The fix was a bypass.
skip the degenerate Thinking Process loop and answer directlyClaude Fable 5.1, final notes, session 1
What I take from this
The method works in the sense that matters for the report. A frontier model read activations, found a handle four neurons wide, and changed held-out behaviour with no gradient. The random-location control says it was not luck.
But twice the editor produced a specific, measured, mechanistic account of the loop, with named heads, named neurons and effect sizes in nats, and a precise edit that followed from it. Twice the edit did nothing. The explanation that worked was not about the loop. It was about the door into the room where the loop happens, and the editor closed the door.
"A frontier model can find a small lever that changes the behaviour" is shown. "A frontier model can understand why a small model misbehaves and fix the cause" is not, at least not here. The MIT edit, by the same count, is a repair. So the open question is what made the difference: the editor, or the task.
Next
Two runs with GPT-6 Astra, the report's own editor, on this SQL task. An ablation map, switching off every head and every block of neurons one at a time, to learn whether a single-site repair exists at all. And a robustness check borrowed from MaxxBench: run the edits on Dr.Spider, which pairs each Spider question with a minimally changed twin. A bypass should do no better there than the flag. A repair might.
Everything is released: both edit files (19 KB and 10 KB), every turn of both sessions, the editor's notes, the harness and the evaluation scripts. Applying an edit and rerunning the BIRD numbers takes about 20 minutes on a laptop. The arXiv version follows once the Astra runs are in. The neuron view is inspired by Levi Foster's circuit grafts, which add new neurons to a 135M model instead of changing old ones.