“Super Intelligence” Evaluation Battery
SILT Newsletter #047
#047

A number we quoted could not be re-derived, and neither could our description of it

Three trim-survival shares for the Code Integrity Battery judge panel were computed once and written into the record. When a page tried to derive them rather than copy them, they did not return. Our own write-up of the finding carried the same error. Both were corrected on 25 September 2026.
◆ In the News▲ SILT Analysis & Response● What We're Watching
01In the News
We publish, for the Code Integrity Battery, how often each judge is the one the trimmed mean drops. Three such shares were computed once, on 808 cells, and written into the record as settled fact. Nothing recomputed them after that. When a page tried to derive those shares rather than copy them, they did not come back. Under two defensible definitions they differ by up to 36 points on one judge. The cause is structural: the trimmed mean sorts scores, not judges. When judges tie at the highest or lowest score, which judge was dropped is not a fact about anything. Re-derived on 25 September 2026 over 859 complete panels, 51% of panels have at least two judges tied at an extreme, and only 49% have a unique top and bottom. We correct this in the open. The error was found only because we re-derived the figure for this issue rather than quoting it again.
02SILT Analysis & Response
The second act is ours. Our write-up of that finding reached the public C.I.B. methodology page, and it said about a third of panels were unanimous, all four judges giving the identical score. The count was right. The description was not. That third is 301 panels, 35%, where every judge sits at an extreme, mostly two-and-two or three-and-one splits. Truly unanimous panels, all four on the same score, number 16 of 859, which is 1.9%. We corrected the page on 25 September 2026. The argument survives on the right number. The judge trimmed least is still the most consistent one. What does not survive is quoting the shares as fixed values, because they depend on a tiebreak the method never defines. A figure that is quoted rather than derived cannot be wrong out loud, which is precisely the problem: it stays wrong silently until someone recomputes it.
03What We're Watching
The rule we are adopting is to derive a number on every use rather than writing it down. Severity and judge spread carry no tiebreak and are re-derived every run, so they are not exposed to this failure. The mean judge spread stands at 2.61 across 2165 scored cells. We are auditing the rest of the published record for figures that were computed once and never recomputed. The question is not whether each number was right when written. It is whether anything would notice if it stopped being right. A share that depends on an undefined tiebreak is the clearest case, but any quoted-not-derived figure carries the same risk. We will report what the audit finds, including further faults of our own.

All issues: SILT Newsletter. Return to Portal home.