VisuallyMeta
← Field Notes

Correct Judgement Heuristics

Part of Automated Quality Management

When I made "not applicable" a real checkpoint state — alongside pass and fail — it quietly broke the scoring math.

Say a parameter has five checkpoints and one genuinely doesn't apply to this call. Score it out of 5 and you've penalized the agent for something that was never in scope. The honest denominator is 4, not 5. "Not applicable" isn't a silent zero — it's an exclusion, and exclusions change the denominator.

So the logic changed: N/A checkpoints drop out of the denominator entirely, rather than counting as fails. The parameter scores 1 of 4. The header reads "14 of 17 scored," with the three excluded checkpoints shown explicitly — not hidden to make the number look cleaner.

That visibility is the entire point. An agent disputing the grade can walk the exact chain: this checkpoint was N/A, here's why, that's why the parameter is out of 4, that's why your total lands where it does.

The correction wasn't hard to make. It was hard to notice — the math looks correct right up until you realize a third state was silently doing a fail's job in the denominator.