Comparison
Golden testvsRegression testing
Golden test
you save the current output to a file and the test just checks nothing changed, which is exactly what you want on a legacy refactor.
A test comparing output against a recorded reference. It is excellent for pinning behaviour you are about to refactor and for large outputs no one wants to assert field by field. It goes bad when updating the snapshot becomes reflexive, at which point the test records what the code does rather than what it should.
Full entry →Regression testing
you re-run the whole scored set after a prompt tweak or a model upgrade, so you find out you broke case 12 before your users do.
Re-running the eval set on every prompt, model or pipeline change and comparing to the previous run. The habit that separates teams who know a change hurt quality from teams who find out from customers. Model upgrades are changes; test them like deploys.
Full entry →