HumanEval
codingHumanEval measures code generation ability by asking models to complete Python functions given a docstring description. It consists of 164 hand-crafted programming problems.
View paper / source42
Models Tested
97.0
Best Score
90.0
Average Score
0–100
Scale Range
1x
Weight
How It Works
The model receives a function signature and docstring, then must generate the function body. Each solution is tested against a suite of unit tests. The primary metric is pass@1 — the percentage of problems solved correctly on the first attempt.
Why It Matters
Code generation is one of the most practical and measurable AI capabilities. HumanEval provides a standardised way to compare models on programming tasks that range from simple string manipulation to algorithmic problem-solving.
Limitations
Only tests Python. Problems are relatively simple compared to real software engineering. Models may have memorised solutions from training data. Does not test debugging, code review, or working with existing codebases.