We quantized our AI judge. Here's exactly what broke.

Our production judge — a small 1.7B model with a LoRA adapter that grades other AI outputs as pass / fail / insufficient_evidence (88.5% accuracy, ECE 0.072) — is cheap to run. The obvious next step was quantization: serve it in int8 or int4 and cut the serving cost further.

Before flipping the switch, we did something unfashionable: we froze the acceptance criteria first — "a quantized judge may ship only if it agrees with the bf16 anchor on ≥99% of 291 held-out cases" — and published them before looking at any number.

Results (all measured, artifacts sha-anchored):

Drift grows monotonically with quantization strength. The two int8 flips and seventeen int4 flips are each a case where the judge changed its verdict — not its confidence, its answer. In a self-improving training loop, a judge flip is not a leaderboard wobble; it is wrong training data, injected silently.

The four-rule deployment discipline

1. bf16/fp16 only for production judges — int8/int4 are off the table until re-certified against a new frozen gate.
2. Freeze criteria before running. Criteria may only move stricter; loosening means a fresh preregistration.
3. Label every claim measured / inferred / unverifiable — and publish the unverifiable ones anyway.
4. Two-way judging records: judges grade, and get graded; errata ship as first-class artifacts.

The full study

Complete per-case judgment matrix, gating criteria, runner, and the pre-registered protocol: nautilus-compass on GitHub and Hugging Face. The arXiv version ships this week.