Weight-space programming, measured in bits

We wrote a long system prompt into a language model’s weights, with no training run, and then deleted the prompt. On 256 reviews it had never seen, the model still scores 0.951 on the task; the original, reading the full prompt, scores 0.979. The same approach made Othello-GPT win more games with 92 added neurons, and lifted a coding model on MBPP+ from 59.5% to 64.6%, where fine-tuning on the same data made it worse. Each step was guided by one measurement. bitsy, our open-source library, asks how much a model learned from a prompt, a fine-tune or a weight edit, and answers in a single unit: the bits it saves on the right answers to held-out questions.

Last week a group at MIT CSAIL showed that a frontier language model can read a network’s raw weights and rewrite them with no training loop [2]. It worked out which game a policy network had learned, fixed a car controller by changing 3.5% of its weights, and lifted Qwen3.5-0.8B from 7.9% to 48.1% on MATH-500. Once a model can write weights, what matters is measuring three things: how much an edit installed, what else it changed, and whether the gain holds on data the editor has never seen. One task score cannot separate them.

Three experiments

bitsy measures the first two in bits. What an edit installed is the bits it saves on held-out answers; what it disturbed is the bits it adds on ordinary text. For the third, it uses a sequential test that compares two models one fresh example at a time, with an error rate that holds however often you check [1]. Claude (Opus 5.5) proposes each edit and computes it in closed form; bitsy measures the result. We tried this on three problems.

The first is compiling a system prompt into a model’s weights. ReviewGuard is a long system prompt that tells Qwen2.5-1.5B-Instruct how to turn a customer review into a star rating, a sentiment and a one-sentence summary. We wanted a model that behaves as if it had read the prompt without the prompt being there, so nobody pays to process it on every call.

The second is making Othello-GPT play better. Othello is played on an 8×8 board: each move places a disc and flips every opponent disc it traps in a line, and whoever owns more discs at the end wins. A corner disc can never be flipped, so corners are valuable. Othello-GPT is a small transformer trained only to predict the next move from game records. It never sees the rules or the board, yet it builds an internal picture of the board [3]. We wanted it to win more games through edits alone, so each new version, which we call a generation, had to beat the current champion on fresh games before it could replace it.

The third is making a small coding model write better code. Qwen2.5-Coder-1.5B-Instruct passes 65.2% of HumanEval+ and 59.5% of MBPP+, two standard Python benchmarks with extra tests. We wanted it to pass more of them without forgetting anything else.

In every case Claude solved the edits in closed form from the model’s own activations, in the spirit of MEMIT [4], so the edited model never took a gradient step.

What bits measure, and what to measure them on

A model that gives probability p to the token that actually comes next needs −log₂ p bits to encode it. Summed over a text, that is the model’s code length; per token, it is log-loss in base 2. What a prompt, a fine-tune or an edit teaches a model is the bits it saves on held-out answers, compared with the model without it. bitsy works in bits rather than accuracy for two reasons. Bits move smoothly as a model gets closer to the behaviour you want, whereas accuracy stays flat until an answer flips. And code lengths can be compared and tested sequentially, which the next section relies on.

Bits are only as useful as the target you measure them on, though, and Othello made this very clear to us. We measured bits on three targets, and only two of them helped (Table 1).

Human moves are a poor target because they include mistakes and personal style. Log-loss on them rewards a model for copying people and for being well calibrated, not for choosing better moves. Our first edit cut legality bits by 28%. It also “improved” imitation bits by 0.8, mostly by flattening the model’s distribution, yet it played slightly worse than the original.

Legality is a good target because the input determines it completely. The moves played so far fix the board, and the board fixes the legal moves. A perfect model therefore pays nothing for legality, and every excess bit is information the model had and failed to use. Measuring where those bits are lost tells you where to look. It also works as a guard. When an early version of generation 2 quietly added 0.065 bits of legality loss, this measurement caught it, and it then confirmed the repaired version (down from 0.441 bits to 0.416).

Deciding on fresh data

To decide whether an edit helped, bitsy compares two models on the same stream of fresh examples, one at a time. It keeps a running measure of evidence, W_t, that model A does better than model B, and updates it after each example:

Here d_t is how much better A did on example t, c bounds what a single example can contribute, and the weight λ_t, at most 0.75, is set from earlier examples only. If A is in fact no better, the evidence cannot grow on average, and Ville’s inequality bounds the chance that it ever reaches 1/α by α. (W_t is an e-value, as in testing by betting.) So the test can be checked after every example and stopped as soon as the evidence crosses 1/α, without inflating the error rate. An editing loop needs exactly this, because an editor that checks the score after every attempt is the case in which ordinary p-values break.

Three details that matter for editing

A non-inferiority margin changes the question to whether the edit is no worse than the prompt by more than, say, 0.05. That’s the right question when the aim is to replace a prompt rather than beat it. Bits are mixed with a little of the uniform distribution before testing, which caps what one overconfident token can cost, and the mixing costs at most 0.015 bits per token. And the test accepts any bounded score, so it can run on graded answers or game results as well as bits.

In the paper, we ran 32 comparisons between GPT-2 reading a cached context and GPT-2 fine-tuned on the same data. The test picked the better of the two on 61,948 of 64,000 query streams.

The editing loop

Claude proposes an edit and bitsy measures it on practice data. The edit is kept only if it passes the test on data the editor has never seen.

Compiling a system prompt into the weights

With no prompt at all, the compiled ReviewGuard model keeps 97% of the prompted model’s score on unseen reviews, and adds under 0.02 bits per token on ordinary text. For a prompt, bitsy’s measure is the number of bits per token the edited model, given no prompt, needs to reproduce the prompted model’s answers. The gap between that and the prompted model’s own bits is what the edit still has to install.

We developed the method on a second prompt, triage, which sorts support tickets into intents. Claude began with single-layer edits. Edits at layer 14 and at layer 20 both scored 0.00 on the graded task, so the task score could not tell them apart, but the bits could (Figure 4). Without the prompt, the model needed 3.23 bits per token to reproduce the prompted answers. The layer-14 edit left that unchanged, while the layer-20 edit cut it to 1.33, about 65% of the way to the prompted model’s 0.31. The task score behaved like a light switch and the bits like a dimmer. Editing the late layers in sequence then reached 0.37.

That edit had a cost the task score didn’t show. On ordinary text (WikiText) it needed 5.04 bits per token instead of 3.25. bitsy’s layer-by-layer readout put the damage in layers 25 to 27, so the fix was to hold ordinary text in place. A three-pass edit that restored ordinary text after installing the prompt cost only 0.036 extra bits per token on WikiText. On 512 held-out tickets it scored 0.901 against the prompted model’s 0.882, and the test found it no worse than the prompted model by more than 0.05. The final method holds the model’s activations on ordinary text in place from the start.

We then applied the final method to ReviewGuard and tested the result on 256 reviews that played no part in building the edit (Table 2).

The upper confidence bounds on the extra bits are 0.025 for web text (C4) and 0.016 for chat (Dolly). The test concluded after 190 reviews that the compiled model is no worse than the prompted one by more than 0.05. 98.4% of the compiled model’s outputs were in the required format.

On a separate set of 256 tickets, the final method’s result on triage was inconclusive: its compiled model scored 0.879 against the prompted model’s 0.900, and the test had not concluded either way, so we make no claim for it. The earlier three-pass triage edit passed.

In a separate run on ReviewGuard we compared the compiled edit with LoRA fine-tuning (Figure 5). At the same scope and rank, the compiled edit scores 0.969 against LoRA’s 0.946, and adds 0.027 bits per token on web text against LoRA’s 0.086. The best LoRA, rank 16 on all layers, scores 0.978 but adds 0.081 bits per token, three times as much, and needs a training run.

Improving Othello-GPT one generation at a time

Two generations of edits, 92 added neurons in all, made Othello-GPT stronger twice in a row: +33 Elo, then +16. Each Othello edit is written in closed form from probes of the model’s own state and from its output weights. We then play the edited model against the current champion on fresh, balanced openings. Each opening is played twice, with colours swapped, while guards check that legality bits and the board probes have not got worse. Each generation’s test gets a smaller α, so the chance that any conclusion along the way is wrong stays small. For the two generations so far, it is under 0.5%.

What the model knows, and what it does

A linear probe on the model’s activations can tell whether a corner is available with an AUROC of 0.99. It can tell whether a move would hand the opponent a corner with an AUROC of 0.93. Yet the model takes an available corner only 45% of the time, where human tournament players take 50%. That gap between what the model knows and what it does is where we edited, and games decided whether each edit helped.

Generation 1

Generation 1 added 8 neurons in block 6 that read the model’s own corner probe and push it towards available corners. It scored 54.7% of points against the original, and the test concluded it was a win at α = 0.0034.

Generation 2

Generation 2 added two edits. 24 neurons in block 6 stop the model handing corners to the opponent. 60 neurons in block 7 put less probability on illegal moves, which repaired the legality cost described earlier. The test concluded after 3,581 pairs of games at α = 0.0012. Over all 4,926 fresh pairs played against generation 1, generation 2 scored 52.3% of points. It won 1,334 pairs and lost 1,078; a pair is won by taking more than half its points.

Games have to make the final decision. A gradient-based edit aimed at the same legality measure lowered its legality bits by piling probability onto legal moves. It then lost 1,005 of 1,265 opening pairs to the original.

Making a coding model better

The obvious way to improve a coding model is to fine-tune it on correct solutions to problems it gets wrong. We did that for one epoch with verified solutions to 640 practice problems, and it made Qwen2.5-Coder-1.5B-Instruct worse: 61.6% on HumanEval+ and 57.4% on MBPP+, down from 65.2% and 59.5%.

Measuring in bits showed why. We scored the correct solutions and also the model’s own current answers. A two-epoch fine-tune improved the correct solutions by only 0.026 bits per token but made the model’s own answers 0.19 bits per token worse. Most of what it learned was to move away from its own outputs, and it kept only 76% of the held-out practice problems the model already passed.

The fine-tune as a map

So we used the fine-tune as a map of what the right answers need, not as the answer. Bits chose which copy to use. In the second epoch, training loss kept falling but bits on held-out correct answers went back up, so we took the one-epoch model (Table 3).

Bits also chose where to edit. We moved one layer’s share of the fine-tune’s change into the original model at a time and measured held-out bits (Figure 7). Removing any single layer from the fine-tune never hurt, so most of its change was redundant.

The edit

Claude then solved for a change to one matrix per layer, the MLP output projection, working from layer 6 to layer 18. For each layer it ran the model, with the earlier layers already edited, on 256 practice problems with their correct solutions. At every token, the target is the difference between the fine-tune’s MLP output and the original’s. On 884 other texts the target is zero: 500 of the model’s own correct answers, 256 windows of WikiText and non-Python code, and 128 general chats. One least-squares solve per layer balances the two:

Here t is the target change and k is the input to the output projection. The second sum runs over the training and held texts together, and λ is 10⁻⁴ of its mean diagonal. Each of the 13 matrices moved by 1.1–1.6% of its norm. In bf16 that changes 149 million numbers, about a tenth of the model, as a dense change merged into the weights with no adapter. The whole solve took about six minutes on one A100.

The edit reproduces about two-thirds of the fine-tune’s change on the training problems. It leaves out the fine-tune’s changes to attention, to the other layers and to the MLP inputs, which is where most of its damage was.

Results

The fine-tune on its own scored below the original on both benchmarks, while the edit scored above it: it keeps the useful part of the fine-tune’s change and leaves out the rest. Across both benchmarks the edit fixed 40 problems and broke 16 (p = 0.002, or 0.007 after correcting for the four variants we tried). MBPP+ improved significantly on its own (p = 0.004); HumanEval+ did not (p = 0.30). Ordinary text and MMLU didn’t move. On practice problems held out from the solve, the edit breaks 16 of the 192 the model used to pass and fixes 26 of the 192 it used to fail.

Where bits didn’t help

Bits didn’t pick the winner. Copying the fine-tune’s MLP weights for layers 6–18 straight into the model looked good on log loss, but it broke 32 of 192 problems the model used to pass and gained nothing on the benchmarks. Pass rates under the problems’ unit tests chose the final edit. Its gain over the previous version came from two changes to the solve: holding more of the model’s own answers in place (about 500 instead of 128), and weighting the training problems equally with them rather than three times as heavily. Both are constraints in the solve, not measurements.

The layer-by-layer readout that located the damage in the prompt experiment found nothing here: its curves for the original and the fine-tune looked almost the same. Choosing MLPs over attention was a judgement call. All the attention layers together saved more bits than all the MLPs together (0.093 against 0.066 bits per token), but the MLP effect is concentrated in fewer layers, so fewer places needed editing.

What this shows

The prompt and Othello edits made a model act on something it already had: a behaviour it shows when prompted, or knowledge it holds but doesn’t use. That matches the MIT finding that editing can remove an obstacle but can’t add an ability the model lacks. The coding edit is a variation on the theme: it kept the part of a fine-tune that fixed problems and left out the part that broke them.

In all three, one measurement guided the work. Bits showed how much a prompt, a fine-tune or an edit had taught the model, where the next edit should go and what it disturbed; games and tests on fresh data decided whether it stayed. That is the part MIT’s results leave open, and the part bitsy is built for.

Limits

All three models are small. With the final method, only one of the two prompts passed the test; triage passed only with the earlier three-pass edit. Generation 2 hasn’t yet played the original directly, so the +49 in Figure 6 adds two head-to-head results rather than measuring one. The coding gains are modest, HumanEval+ alone isn’t significant, and that edit was solved from a gradient-trained reference, so it is training-free only in the sense that the edited model never takes a gradient step. None of this has been reproduced independently yet.

bitsy is open source at github.com/leochlon/bitsy, and the theory behind it is in our paper, arXiv:2610.29419.