How AI Benchmarks Work
(And Why You Should Care)

Every time a new AI model launches, the press release is filled with numbers: "92.3% on MMLU", "1370 ELO on Chatbot Arena", "78% on SWE-bench". But what do these numbers actually mean? And should you trust them?

What Are AI Benchmarks?

AI benchmarks are standardised tests that measure how well a model performs at specific tasks. Think of them like exams for AI — each benchmark tests a different skill, from general knowledge to code generation to following instructions.

Just as you wouldn't judge a student by a single exam, you shouldn't judge an AI model by a single benchmark. The best approach is to look at performance across multiple benchmarks that test different capabilities.

The Key Benchmarks Explained

Knowledge: MMLU & MMLU-Pro

MMLU (Massive Multitask Language Understanding) is the "general knowledge exam" of AI. It tests models across 57 subjects — everything from abstract algebra to world religions. Each question is multiple choice with 4 options.

The problem? Top models now score over 90%, so MMLU is getting less useful for telling the best apart. That's why MMLU-Pro was created — it uses 10 answer choices instead of 4 and harder questions.

Reasoning: GPQA Diamond & MATH-500

GPQA Diamond is the hardest academic benchmark. Questions are so difficult that PhD experts in the relevant field get less than 65% right. If a model scores well here, it genuinely understands deep scientific concepts.

MATH-500 tests competition-level mathematics — problems from AMC, AIME, and other maths competitions. These require multi-step logical reasoning that can't be solved by pattern matching.

Coding: HumanEval & SWE-bench

HumanEval tests whether a model can write correct Python functions. It's 164 problems with test cases — the model either passes the tests or doesn't.

SWE-bench Verified is much harder. It gives the model a real GitHub issue and an entire codebase, and asks it to produce a working fix. This is the gold standard for "can this AI actually do real software engineering?"

The People's Choice: Chatbot Arena

Chatbot Arena by LMSYS is different from all the others. Instead of automated tests, real users have conversations with two anonymous models side by side and vote for which one they prefer. The results are compiled into an ELO rating (like chess rankings).

With over 2 million human votes, it's widely considered the most reliable indicator of overall model quality. It captures things that automated benchmarks can't — like writing style, helpfulness, and common sense.

Why Benchmarks Aren't Perfect

Every benchmark has limitations:

• Data contamination: Models may have seen benchmark questions during training, inflating scores.
• Narrow testing: Each benchmark tests a specific skill. High scores don't guarantee good performance on your specific use case.
• Gaming: Companies can optimise models to score well on specific benchmarks without improving general capability.
• Saturation: When all top models score 95%+, the benchmark stops being useful for comparison.

How to Use Benchmarks Wisely

1. Look at multiple benchmarks — no single number tells the whole story.
2. Prioritise benchmarks relevant to your use case — if you need a coding assistant, SWE-bench matters more than MMLU.
3. Check Chatbot Arena — it's the closest thing to "which model do real people actually prefer?"
4. Try the models yourself — benchmarks are a starting point, not the final answer.