Comparison
BenchmarkvsEval
Benchmark
you read that a model scores ninety on a public suite, try it on your own task, and find the number told you almost nothing.
A public, standardised eval suite: MMLU (broad multiple-choice knowledge), HumanEval (short Python problems), SWE-bench (resolving real GitHub issues), MTEB (embeddings), among many. Useful for rough model comparison; none of them measures your task, which is why you build your own set.
Full entry →Eval
you stop saying the answers feel better and produce a number you can put next to last week's number.
A measurement of model or system behaviour against defined cases and scoring, producing a number you can track. The word covers everything from 20 assertions in a script to formal benchmark suites.
Full entry →