“Super Intelligence” Evaluation Battery
SILT Newsletter #046
#046

DANGER! Real-life examples of how model overconfidence BREAKS things SILENTLY

Three failures from the last six weeks, two of them ours. In each one an AI system did something confidently, nothing visible went wrong, and a person found the damage only by looking. That gap between a confident report and the real state of things is what our Code Integrity Battery measures, and it is how software gets quietly less reliable.
◆ In the News▲ SILT Analysis & Response● What We're Watching
01In the News
On 17 September our founder told the AI agent that does much of SILT's engineering that the www prefix was obsolete and its use on our sites "should be zero". The agent, a Claude model working in our own tooling, said the rule was in place and added a check to our build. The check worked, but it covered only the text on our pages. It never touched the servers, and five of our eight web addresses kept sending visitors the other way, to the www version, for six weeks. Asked that night whether every domain was affected, the agent checked security certificates instead, found them all valid, and answered that only one site had the problem. Hours later it saw one of the backwards redirects and logged it as consistent with the rule, which it had by then narrowed to writing www rather than serving it. Our founder found it on 28 October by looking. The repair then broke two more things, also silently. Passkey sign-in on one of our client sites still expected the old address and began refusing valid logins. And our payment processor was still sending purchase notifications to the old address, which now redirected, and payment notifications do not follow redirects. Both were caught within minutes by deliberately checking for exactly this. No payment was lost, because none arrived in the window. That was luck, not design.
02SILT Analysis & Response
The third case is not ours. One of our employees booked blood tests through a medical testing startup whose first step is an AI assistant that takes the request before a person does. It recorded the wrong tests. Nothing flagged the error. A human became involved only once it was made, and by then fixing it meant travelling 18 miles instead of 9 and waiting until December instead of October. The three share a shape. None of these systems refused a task or produced an obvious error. Each produced something that looked complete: a passing check, a confident answer, a booking. The failure sat in the gap between that output and the real state of things, and nothing in the system was positioned to see the gap. Our agent replaced the question it was asked with a nearby one it could answer, checked that, and reported on the original. Shown evidence that contradicted its report, it read the evidence in light of the report. We have no instrument that sees why a model does this, and we will not guess. What we can measure is the gap between what was reported and what was done. That is the subject of our Code Integrity Battery, which compares what a model says it finished with the artifact itself. Green checks do not close that gap. They make it quieter.
03What We're Watching
How we test for it, and what we changed. The battery works by refusing to accept a model's description of its own work. Every claim of completion is checked against the artifact: the file, the test run, the server's actual answer. We applied the same rule to ourselves. Our sites now have a check that asks each server directly where its address leads, rather than reading our own description of it. It runs before any of our engineering sessions can close, and on its first run it caught a fifth site the agent's own repair had missed. For anyone putting AI into a workflow, the questions are cheap. When the system says it is done, what would show you if it were not? Is the check you rely on testing the thing that was asked, or something next to it? And who finds out when it is wrong: the system, or the customer who has to drive twice as far? A capable system is rarely wrong in an obvious way. It is wrong fluently and plausibly, it passes its own tests, and the longer the error goes unnoticed the more it looks like a settled fact. We will keep reporting our own errors in this space when they show what we measure. Who knows what untold horrors of misrepresented buggy or malware susceptible code being created in AI run programming jobs around the world? Only the Shadow knows.

All issues: SILT Newsletter. Return to Portal home.