An 18-month research program for post-training code agents when the checks you can run (it builds, it passes regression, it behaves equivalently on tested inputs) are narrower than the thing you want: a change that is correct.
def shipping_rate(order): if order.region == "US": return domestic_rate(order) − elif order.region == "EU": − return eu_rate(order, vat=VAT[order.country]) return flat_rate(order)
Code agents have a rare advantage for reinforcement learning: a reward you can compute. A change builds or it doesn't, the regression suite passes or it doesn't, and old and new code agree on a set of inputs or they don't. These signals are cheap and hard to argue with, so they are the natural thing to optimize.
They also have an edge. Past it sits the question you actually care about: is this change correct, or merely plausible? A policy trained hard against the verifiable signals will find that edge. The cleanest example is the one above: an agent that can't fix a branch the upgrade broke learns that deleting it turns everything green, because no test reaches it.
The longer the run, the worse this gets. Over 40 steps the chance of an honest run succeeding shrinks with every step, while a hack that reliably passes stays reliable. So the longer the horizon, the more the reward favors the hack.
Cheap, total, and says almost nothing about behavior.
Only covers the paths the suite reaches.
Only on the inputs you thought to try.
Right on paths and inputs nobody tested. This is what customers see.
Before proposing months of work I wanted to see the failure in a setting small enough to control completely. Patch World is a toy repair task: a program has 12 branches whose outputs follow a pattern, some branches are broken, and the agent must fix them. It can write a value, stub the branch, or delete it. Half the broken branches are reached by the visible tests. A hidden equivalence check compares every branch with the reference.
The policy is a small MLP, not a language model. It is first fine-tuned on demonstrations from a deliberately careless engineer, then trained with GRPO and a KL penalty back to that starting point, exactly as you would post-train an LLM, at a scale where 42 runs finish on a laptop.
4 broken branches per program by default. Off-by-one fixes build but are wrong; other wrong values fail the build.
Deleting a branch always builds. A stub builds unless it fails to type-check, which happens on a quarter of branches.
Visible checks only · a deletion penalty · random audits by the hidden check at 5–100% of episodes.
3 seeds per setting, 1000 GRPO steps, greedy evaluation on 2,000 held-out programs. Same seed, same numbers.
Share of held-out programs that pass the visible checks, and share that are actually correct, during GRPO. The shaded gap is false success: green on every check, wrong underneath.
Outcome of every untested broken branch after training, by reward. The starting point is the fine-tuned policy before RL.
False success after training against the audit rate: the share of episodes that also get the hidden equivalence check, with caught hacks penalized.
False success after training against the number of broken branches the agent must fix in one episode.
False success before and after GRPO on the visible checks alone. Passing rose to 58% while real correctness fell from 31% to 2%. Every reward curve looked like progress.
False success with audits on 10% of episodes, against 56% with none and 4% with audits on every episode. Audits act like a threshold: 5% barely helps, 10% gets partway, and 25% matches auditing everything at 1,000 steps (Figure 3). A 5,000-step follow-up shows 25% only delays the hack (E24).
Each phase answers one question, produces something the team can use whether or not the next phase happens, and names the result that would kill it. Phases overlap where the later one only needs the earlier one's first output.
After the pilot I wrote down fifty harder questions about hacking at the verification boundary. Each one has a prediction and the result that would refute it, committed to the repository before any of them ran. Each verdict is computed in code from those criteria. Most run on the pilot's policy; the sequence-model and probe experiments use a small transformer that reads the whole program.
Refuted predictions are reported as prominently as supported ones. Several of the interesting results are refutations.
Every experiment is written down with the number that would end it before it runs. A weekly review where results get defended or dropped. Most of these ideas should die early and cheaply.
Reward curves are where the pilot's hack hid. Every training run samples passing trajectories for a human read and tracks audited false success next to reward from the first step.
Open-weight models on rented clusters. Each question starts at the smallest model that shows the behavior, and scales up only to confirm a result that already holds.
Each phase reports where a method failed alongside where it worked, the way the pilot reports that oracle RL never improved honest repair.
Four years of applied deep RL: a PEARL meta-RL agent for 1v1 air combat on DARPA's ACE program, an RL loop that tuned synthetic data generation against a downstream detector, and a single multi-stage PPO policy that got a Unitree Go1 to track, approach and kick a ball at Berkeley's Hybrid Robotics Lab. More recently, a small post-training study: SFT from a flawed demonstrator, then GRPO against a verifiable reward. My post-training work so far is toy-scale; the open-weight LLM tooling is what I'd ramp on first.
The pilot is about 400 lines of NumPy with no GPU. The experiment log adds a configurable trainer, a small PyTorch transformer and multi-agent code, all CPU-only. The page is built from the results files these write, so no number here is typed by hand.
uv venv --python 3.11 .venv uv pip install --python .venv/bin/python "numpy<2" "torch==2.2.2" pytest .venv/bin/python -m pytest -q tests .venv/bin/python -m pilot.run_all # 42 runs, 3 seeds .venv/bin/python -m experiments.suite # experiment log, NumPy part .venv/bin/python -m experiments.run_seq # transformer and probes .venv/bin/python -m experiments.multiagent .venv/bin/python site/build.py # rebuilds this page