Applied AI·Evaluation and reliability
your judge kept preferring whichever answer was shown first, so you ran each pair both ways round and averaged the scores.
Judge biases
Draft summary, pending review
Known distortions in model judges: position bias (favouring the first answer shown), verbosity bias (favouring longer answers), and self-preference (favouring outputs in its own style). Mitigations: swap answer order and average, control for length, use a judge from a different family.