Field note 05
September 8, 2026
Fluent and Wrong
An AI reviewer read the same work as a set of executable tests. It called most of the broken work excellent, and its blind spots have a shape.
Every argument for keeping a Golden Thread was an argument about people. Humans read the registry, humans follow the pointers, humans stay oriented. Fine. But most code now is written by agents, so the uncomfortable question is whether the thread ever helps the machine, or whether all of the help is for us.
That is a question you can answer badly in an afternoon or honestly in two days. SpecAssay took the second route, and the finding that matters most is not the one the trial went looking for.
The design, in plain terms
One Java codebase, cut into three copies that differ only in how much thread is present. One copy has the full apparatus. One has informal prose pointers to specs and nothing machine-checkable. One has neither.
The acceptance criteria were turned into executable tests first, frozen, and never shown to any agent in any arm. Cold agents got an ordinary plain-prose ticket that said nothing about registries, IDs, or traceability. One hundred sixty-three of them ran.
Every result was graded twice: once by those hidden tests, and once by a blind AI reviewer reading only a scrubbed diff. Predictions were written down before the first run.
The second grader was meant to be a convenience. It became the finding.
The number the trial went looking for
On a task turning on what the business had actually promised, the threaded arm was right every time and the thread-free arm was right less than half the time.
| what the agent had | acceptance score, 8 runs each |
|---|---|
| the full thread | 1.000 |
| informal prose pointers only | 0.667 |
| no thread at all | 0.396 |
Complete separation: every threaded run beat every thread-free run. The gap is not luck; it survives correction for having run twelve comparisons.
Now the sentence that has to follow it. Of five scoreable tasks, one moved. The effect is real and it is narrow, and it lives exactly where an answer has to be looked up rather than worked out. Refactoring work showed nothing. Cleanup work showed nothing.
The companion result is the one that changed how the room thinks about agents. Across one hundred twenty runs, agents opened the spec eight times out of eight when the ticket turned on something that had been promised, and zero times out of eight when it did not, in every arm. Agents do not read documentation because it is there. They read it when the work turns on something only the documentation knows, and they are unnervingly good at telling the difference.
What happens when there is nothing to look up
An agent asked for a rule with a specific threshold, and given no way to find the promised one, does not stop and ask.
It looks around, finds a nearby rule that resembles the request, and quietly adopts that rule's numbers. Every thread-free run on one task took a neighboring rule's window of "more than 2 in the past 3 years" instead of the promised "2 or more during the current term," and every one reused an existing 45-day window where 60 was promised.
With no neighbor to borrow from, it invents. On a task requiring an exact piece of customer-facing wording, not one thread-free run produced the pinned phrase. All of them wrote reasonable English that was not what the business had committed to.
Borrowed or invented, the result is the same: work that compiles, passes its own tests, reads well in review, and is wrong in precisely the place the business cared about.
Nobody wrote a bug. Somebody answered a question they could not look up. That is gilt, arriving by a route nobody was watching.
The finding nobody went looking for
Across the eighty-four runs graded by both instruments, the AI reviewer's scores and the executable tests' scores were uncorrelated. Pearson r was minus 0.031. Not weakly related. Unrelated.
- Runs the tests called broken: 40
- Of those, runs the reviewer called excellent: 29 (72.5%)
- Share of everything it graded that was broken work scored near-perfect: 29 of 84 (35%)
These were not close calls. Of those twenty-nine, twenty-two failed exactly three acceptance criteria and six failed four, while most of the runs it scored correctly failed none at all.
Where the reviewer is deaf
The useful question is not how often a reviewer fails. It is whether the failures cluster, because if they cluster you know where a deterministic check has to stand.
They cluster, and not where you would guess. Not by condition: the reviewer was about equally unreliable on threaded work and thread-free work. Good documentation does not make a model better at reading your diff. It clusters by the kind of defect, hard, from every miss to none:
- A scheduled notice that never fires at all: missed 17 of 17
- A required referral simply absent: missed 6 of 6
- Behavior present, but firing always instead of only in the right case: missed 11 of 13
- State correct in memory, lost on save: missed 11 of 13
- The wrong branch taken, declining where it should have issued: missed 0 of 16
- The wrong string where exact wording was pinned: missed 0 of 16
Read it twice and the rule falls out. The reviewer sees what is written and cannot see what is missing, or what only exists at runtime. A wrong word is in the diff. A wrong branch is in the diff. A job that never runs, a check that never applies, a field that never survives a save: none of those have a diff hunk, and a reader looking at code cannot see an absence.
One admission against interest, because it is the weakest joint in the analysis. Each task contributed one bundle of defect types, so "blind to absences" and "lenient on that particular task" fit the numbers about equally well. Two comparisons inside a single task favor defect type, and both rest on single-digit counts. The sample is too thin to settle it. What is settled: the failures cluster somewhere sharp, and condition is not where.
Why the assay stays deterministic
People ask, reasonably, why Gate 2 does not simply ask an AI whether the work looks right.
This is why. A Gate that asked a model whether the thread held would have waved through most of the broken work in the trial, confidently, in fluent prose. Row states move on parsed marks, named proofs, and exact-set registry checks because those are checkable, and "does this look right" measurably is not. Silent gaps fail deterministically, or they do not fail at all.
The three questions worth spending determinism on are the three a diff cannot answer: did it fire, did it fire only when it should have, and is it still there after a save.
Monday morning
- Take one acceptance criterion that matters and imagine someone deleted the call to the code that answers for it, leaving the code itself untouched. Ask what in your process would notice. A person or a model reading the diff would not: the function is still there, and it still looks right.
- Find one number or one exact wording in your product that came from a business decision. Ask whether an agent could locate it from the codebase alone. If not, that is where the thread earns its keep.
- Add the negative case to your proof. Most tests answer "does it work." The expensive failures live in "does it stay off when it should."
- Do not let a model's review be the thing that can block a merge. It has a real skill and one fatal blind spot: it reads wording and control flow reliably, and it cannot see what is missing. Point it at the copy, the naming, and the branches, and let something deterministic answer for whether the behavior exists.
- Write down what you promised, with a durable ID, before anyone builds it. Agents look it up exactly when it matters, and only if it exists.
Code is cheap. Proof should be too. Fluent is cheapest of all, which is precisely why it cannot be the thing you check.
Method note: one hundred sixty-three agent runs on one Java and Spring codebase with a thirty-six-row registry, run August 31 and September 1, 2026. Every worker run and every review was Claude Sonnet 5, so the reviewer finding is about that model reading diffs rather than AI review in general, and grader and worker sharing a model family is a real confound. These were cold agents taking a single ticket each on a frozen snapshot, which measures pickup and says nothing about what maintaining a thread costs over months. One task in five moved, and quoting the winning task without that sentence would be a misuse of it. Nothing here is a claim about speed, and nothing here is a claim about humans, who were never in doubt and were never tested. The pre-registered predictions, the amendments, the caveats recorded against the experiment itself, and the scripts that regenerate every number above are kept with the record.