Chatbot Arena ELO
conversationalChatbot Arena (by LMSYS) is a crowdsourced evaluation where real users have blind conversations with two anonymous models and vote for which response they prefer. Results are compiled into an ELO rating system.
View paper / source0
Models Tested
0.0
Average Score
800–1400
Scale Range
1.5x
Weight
How It Works
Users interact with two anonymous models side-by-side and pick the better response. Votes are aggregated using the Bradley-Terry model (similar to chess ELO) to produce a ranking. Over 2 million human votes have been collected.
Why It Matters
Chatbot Arena is widely considered the most reliable benchmark for overall model quality because it captures real human preferences across diverse, unconstrained conversations — not just narrow academic tasks.
Limitations
Ratings reflect crowd preferences which may favour style over substance. English-language biased. User base may not be representative of all use cases. Models can be optimised for arena-style short conversations.