Comparison
Confirmation biasvsJudge biases
Confirmation bias
you believed it was the cache and read the logs looking for the cache, and found something.
Seeking and weighting evidence that supports the hypothesis you already hold. It is at its most costly during debugging and incidents, where a plausible early theory determines which logs get read and which get skipped. The practical defence is to state what would disprove the theory before looking, and to hand the theory to someone who has not spent an hour in it.
Full entry →Judge biases
your judge kept preferring whichever answer was shown first, so you ran each pair both ways round and averaged the scores.
Known distortions in model judges: position bias (favouring the first answer shown), verbosity bias (favouring longer answers), and self-preference (favouring outputs in its own style). Mitigations: swap answer order and average, control for length, use a judge from a different family.
Full entry →