Minus Zero: running a spec-and-test codegen loop entirely on local models
I wanted to know whether a small local model, given a tight contract, a written spec plus a real test file, could reliably write correct code. And whether escalating to a bigger local model when it failed made any real difference, or was just an engineering superstition.
The harness is a script called loop.py, run locally against a Radeon 780M (the same one I’ve been tuning for image generation) with no cloud API calls anywhere in the loop. Point it at a task description, a spec file, a test file, and a target source file, and it asks a model (qwen3:8b by default, since it’s fast and cheap to iterate on) to write an implementation. That implementation runs through the project’s real gate: type-check, lint, test. A failure shows the model the exact error and gets another attempt, up to a ceiling of iterations. Still failing at the ceiling, the harness escalates to qwen3.6:27b and tries again from there. Nothing ships without passing the same gate a human contribution would have to clear.
The test subject was Craymel Ball, a small browser recreation of a minigame from an old JRPG, already under active development with a real Vitest suite. That meant an existing, non-trivial spec-and-test contract to hold the model to, and real call sites waiting to use whatever came out the other end. The same game’s sprites became the subject of a separate 3D-mesh pipeline later on.
Reproducing the baseline
Four pure functions in the game, wall-reflection physics, match-result scoring, circle-overlap collision, and a magnetic burst impulse, had already been written by this same loop earlier in the project. Before asking it to do new work, the sanity check was to blank three of them out and regenerate from the existing spec and test files alone, with no memory of the original implementation.
All three came back byte-equivalent or behaviorally identical to what shipped, which meant the harness could be trusted before asking it to do anything new.
The wall
The fourth function, a magnetic burst impulse pushing or pulling a ball away from a player at a given angle, held a bug qwen3:8b couldn’t see across five tries. It kept failing the same assertion no matter how many iterations it got:
FAIL computeBurstImpulse › applies a pure push straight down
expected { x: -5, y: -0 } to deeply equal { x: -5, y: +0 }The model kept handing back impulse math that was arithmetically correct and kept failing anyway, since the failure had nothing to do with the arithmetic. In IEEE-754, -0 and 0 are equal by == but not by identity: Object.is(-0, 0) is false, and Vitest’s toEqual checks identity on each field. A push angled straight down produces a horizontal component of exactly zero, and depending on which direction the trig landed from, that zero could come out signed. It’s a real, if obscure, edge most working programmers never think about twice.
Five iterations, five failures, even with the exact failing diff shown each time. The consistency itself was the signal to escalate.
| Model | Result |
|---|---|
qwen3:8b | 5 / 5 iterations failed |
qwen3.6:27b | passed on iteration 2 |
The 27b model diagnosed the sign issue directly and fixed it on its second attempt, at roughly 200 seconds per iteration against the smaller model’s few seconds.
// original hand-written fix
return {
x: ix + 0,
y: iy + 0,
};// qwen3.6:27b's fix
if (impulseX === 0) impulseX = 0;
if (impulseY === 0) impulseY = 0;Both fixes reassign the literal 0 to force the positive sign bit, one inline on the return and one as a guarded reassignment beforehand. The fix shape suggests the bigger model understood why the sign was wrong.
The escalation attempt itself got derailed twice before it could even run, and neither reason had anything to do with either model. A game running on the same GPU had already eaten the VRAM headroom the local server needed. After closing it, the Vulkan backend came back with fragmented memory from the crash and refused to load anything at all. Killing the offending process and triggering a pre-built scheduled task that resets the GPU driver fixed both.
New ground
With the harness validated, the rest of the work was new. It came from real refactor targets pulled straight out of the running game. render.ts and ai.ts had four pieces of logic worth extracting into their own tested, spec’d modules: a time formatter for the on-screen clock, a facing-direction classifier for sprite selection, a nearest-point search that both the CPU opponent and the ball-tracking code needed, and a shared frame-index formula that two separate animation systems had each written slightly differently. Each got a fresh spec.md and test.ts written from scratch, then handed to the loop.
This round needed no escalation, one to two iterations each on qwen3:8b. Two of the failures along the way turned out to be mine.
One test asserted that two diagonal facing angles should classify identically, on the assumption that Math.cos and Math.sin of those specific angles would be exactly, bit-for-bit equal. Floating-point math lands one ULP apart even where the underlying trigonometry says two angles should tie, and the model’s implementation reflected that. My worked example had assumed otherwise.
A worked example in a spec had the wrong number entirely, a frame-cycle period missing a multiplication by the frame count. That one got caught before the loop ever ran.
Wiring it in
The four new modules still had to replace the inline logic they’d been extracted from before any of this counted. They went into render.ts and ai.ts in place of the wall-clock timer, the sprite direction lookup, the CPU’s target-seeking, and both of the game’s separate frame-cycling call sites, now sharing one formula where there used to be two slightly different ones. Full gate, run clean:
npx vitest run
Test Files 9 passed (9)
Tests 63 passed (63)The game itself still needed a check in the browser: new game start, CPU tracking a ball correctly, sprite facing changing with movement, timer counting down. A green test suite confirms the code is correct against the contract.
Across two local models and eight shipped functions, a small model given a real spec and a real test file and nothing else produced correct code reliably. That included reproducing its own earlier output cold, and new extractions it had never seen before. When it failed, the same wrong answer came back for the same reason, iteration after iteration, which is what makes an escalation ladder work as an engineering pattern. The bigger model gets called in on a repeated signature, and its much slower iterations paid for themselves by catching what the smaller model never saw.
The contract cut both ways too. A test suite tight enough to catch a signed-zero bug in a six-line function catches a bad assumption in the spec that wrote it just as easily. Two of the mistakes this loop caught belonged to the person writing the spec. The test suite flagged both regardless of which side made the mistake.
(Update: a later test ran a different kind of local harness on this same box, DeepSeek Harness’s round-based create_goal tool standing in for this loop’s escalation ladder. See the write-up.)