When your benchmark saturates, stop ranking and start mining disagreement

This note comes out of Tycho, the program that builds The Ghost. The grading stage measured here is the part of Tycho's pipeline that decides which moments in a transcript count as a real decision.

A pipeline mines steering decisions from AI agent transcripts. Stage 2 grades each candidate moment — is this a genuine steer, and how strong? Fourteen held-out items and six models answered this:

engine	exact	HIT / not-HIT	s/item
Qwen3.6-27B	8/14	10/14	31
Qwen3.8-27B	8/14	9/14	89
Claude Sonnet	8/14	9/14	9
Claude Opus	7/14	8/14	14
Claude Fable	7/14	8/14	13

Everything competent clusters at 8–10 of 14. At n=14 every point is ±7%. This sheet is a terribleness filter; it cannot rank survivors.

Shared misses aren't all model error. A moment models read as a durable delegation policy was labeled MISS because the person was trying out a tool feature. That's context no prompt carries. The sheet is partly saturated.

What replaced ranking

When a sheet can't rank, stop asking it to: filter → disagreement-mine → label the disagreements.

Run two survivors over 300 stage-1 candidates, stratified round-robin by project, with exemplars removed first. Identical prompt across engines. Both engines completed all 300 with zero failures.

unanimous on HIT / not-HIT : 80%  (60 split)
unanimous on 3-way grade   : 73%  (81 split)
"mine it" rate:   Qwen3.6  46%     Sonnet  34%

The confusion matrix:

	strong	weak	none
strong	90	45	3
weak	9	85	7
none	3	14	44

Of 60 binary splits, 48 are "Qwen mines it, Sonnet doesn't." 45 of those sit in one cell: Qwen says strong, Sonnet says weak. Only 6 items in 300 are a strong↔none flip.

The engines are not reading differently. They are cutting the same ordering at different places. Where they disagree is exactly where a human label buys the most information. The question isn't "which model understands the corpus"—it's where does the HIT/WEAK line belong.

Where this goes next

The 60 binary splits are the labelling queue. What it buys: the labels place the line, and whichever engine sits on the right side runs the full 3,800-candidate pass. The question stops being which model is better and becomes where the boundary is.