Evaluation and reliability

you stop saying the answers feel better and produce a number you can put next to last week's number.

Eval

you keep a fixed set of inputs with the answers you already know are right, and every change gets scored against that same set.

Test case / golden set

someone had to sit down and write the correct answers for two hundred cases, and without that your scoring is vibes.

Ground truth / labels

you report that eighty-two percent of cases passed and someone asks whether that was one attempt each or the best of five.

Pass rate / pass@k

the answer is either the right invoice number or it is not, so you assert on it instead of paying a judge model to have an opinion.

Exact match and rule-based scoring

you wrote out the dimensions, the scale and a worked example of each grade, and the judge's scores started tracking yours.

Rubric

you have a second model score the first one's output against a rubric, because there is no exact answer to diff against.

LLM-as-judge

your judge kept preferring whichever answer was shown first, so you ran each pair both ways round and averaged the scores.

Judge biases

a for loop over test cases with assertions and a scoreboard.

Eval harness

you re-run the whole scored set after a prompt tweak or a model upgrade, so you find out you broke case 12 before your users do.

Regression testing

your golden set stayed green and real users still hit a case nobody had imagined, so you started sampling live traffic too.

Offline vs online evaluation

you serve the new prompt to half the traffic and the old one to the other half, and let the outcome metric decide instead of your opinion.

A/B testing

the new prompt goes to two percent of traffic first, and you watch quality and cost before letting the rest through.

Canary deployment

you asked for a citation and got a real-looking one that does not exist, delivered with the same confidence as the parts that were true.

Hallucination

the answer is true in general and nothing in the chunks you actually retrieved says it.

Faithfulness / groundedness

you sent the same prompt twice and got two different answers, and nothing in your code changed between the calls.

Non-determinism

you sample the answer five times and take the majority, paying five times the cost to get the hard cases right.

Self-consistency / majority voting

nothing in your code changed but the outputs are worse than they were in March, because the traffic or the model underneath moved.

Drift

you read that a model scores ninety on a public suite, try it on your own task, and find the number told you almost nothing.

Benchmark

a model posts a startling score on a public test set and does no better than the last one on yours, because the answers were in its training data.

Benchmark contamination

the model at the top of the arena lost to the second one on your task, because humans voting on style is not your workload.

Leaderboards and Elo

the incident came in and you had every prompt, output, model version, latency and cost to look at instead of guesses.

Observability

you record every prompt, tool call and response in one linked timeline, so you can replay exactly what the run did when it went wrong.

Tracing / spans