#045
We measured our own judges
All four seats on the Code Integrity Battery panel score their own family above the peer mean. We publish each effect with its n, disclose a larger severity spread, and correct neither.
◆ In the News▲ SILT Analysis & Response● What We're Watching
01In the News
We ran the Code Integrity Battery's self-preference control over 859 complete-panel cells, derived 25 September 2026, using the same method as the live C.I.B. methodology page. For each cell we took a judge's score on a model from its own family, measured it against the mean of the other three judges on that same cell, then subtracted the same measure on every other model.
The results, by seat: Google +1.15 over 153 own-family cells, OpenAI +0.70 over 158, xAI +0.33 over 74, Anthropic +0.23 over 145. All four signs are positive. That is the part we cannot wave away. The deltas run against the peer mean, so a judge scoring its own family high mechanically drags the other three down on those cells, pushing the set toward negative. Every judge still came out positive anyway.
We publish each effect with its n and deliberately no confidence interval. The instrument already owes a consolidation of its existing estimators. Writing another just to fill a table is an error we decline to commit, so we make no claim about which effects are significant or exclude zero.
02SILT Analysis & Response
A second, larger effect is not self-preference. Severity, each judge's mean deviation from the panel mean, spans 2.30 points on the 1-10 scale, from the Google seat at +1.29 to the OpenAI seat at -1.01. Some judges simply mark harder than others, and that gap is wider than any own-family lean we found.
We decided not to correct the judge score for either effect. A correction inserts an arbitrary adjustment inside a published number where no reader can argue with it. A disclosure lets a hostile reader discount us on their own terms. We prefer the second. When we tested removing severity before trimming, it made the diagnosed problem worse, so the ordering matters and we left it alone.
Scope, so this is not over-read: the judge score feeds the conduct index only. It cannot move the Reliance Gap. We know of no other panel that publishes its own measured biases, which is precisely why we would rather show the numbers than quietly launder them.
03What We're Watching
We are watching the xAI seat, where the +0.33 effect rests on only 74 own-family cells, the smallest base of the four. A thin cell count is not a licence to ignore the sign, but it is a reason to revisit the figure as more complete-panel cells accumulate.
We are also watching the severity spread. A 2.30-point range across seats is large enough that the panel composition on any given cell shapes the outcome. We want to understand whether that range is stable or drifts as models are added and retired.
The open task we have set ourselves is the estimator consolidation we referred to above. Until that is done we will keep reporting these effects with their n and without intervals, and we will keep saying plainly what the numbers do and do not support.