Built-in utilities plus a curated map of the wider AI tool market.

Short-form factuality benchmark for fact-seeking questions with single verifiable answers. Useful for measuring whether a model answers accurately instead of confidently guessing.

Strong fit for the Reliability Floor factuality dimension, but too narrow to stand alone as a general reliability score.

• Targets hallucination on short factual questions
• Includes correct, incorrect, and not-attempted grading
• Useful for calibration and abstention analysis
• OpenAI reports the benchmark was designed to remain challenging for frontier models

Benchmark for whether long-form answers stay faithful to supplied documents while still addressing the user request.

Good fit for the Reliability Floor groundedness dimension because it tests source-grounded answers rather than broad world knowledge alone.

• Long-form grounded response evaluation
• Public and private evaluation sets
• Documents span finance, technology, retail, medicine, and law
• Uses separate eligibility and factual-grounding judgement phases

Verifiable instruction-following benchmark covering constraints such as word counts, required keywords, and formatting rules.

Useful because it avoids purely subjective judging for a core reliability question: did the model do exactly what it was told?

• Around 500 prompts
• 25 verifiable instruction types
• Objective checks for many prompt constraints
• Good first source for instruction-following sub-scores

Long-context benchmark that extends needle-in-haystack into harder retrieval, multi-hop tracing, aggregation, and question-answering tasks.

Use as a retrieval stress test, not as proof of real-world long-context reliability by itself.

• 13 representative long-context tasks
• Tests behavior beyond simple literal retrieval
• Configurable sequence length and task complexity
• Exposes failures as context length increases

Holistic long-context evaluation suite with more application-like tasks including retrieval-augmented generation, citations, summarization, and reranking.

Useful counterweight to simple needle tests because the authors explicitly warn that simple synthetic retrieval does not predict real downstream performance well.

• Evaluates diverse long-context applications
• Controls input length and task complexity
• Includes RAG, citation, summarization, and reranking-style tasks
• Shows categories do not always correlate with each other

Comprehensive RAG benchmark for dynamic factual question answering with web and knowledge-graph search simulations.

Best treated as a RAG-system reliability source, because scores depend on retrieval setup as well as the base model.

• 4,409 factual QA pairs
• Five domains and eight question categories
• Covers popularity and temporal dynamism
• Measures hallucination pressure in retrieval-augmented answers

Benchmark for long-term assistant memory across sustained interactions, including extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.

Useful for the Reliability Floor memory dimension, but memory claims need especially careful evidence labels and limitations.

• 500 curated questions
• Scalable user-assistant chat histories
• Tests five core long-term memory abilities
• Separates memory design choices across indexing, retrieval, and reading stages

Frequently updated objective benchmark covering math, coding, reasoning, language, instruction following, and data analysis.

Useful for fresh-data discipline and objective grading; only the relevant categories should contribute to the Reliability Floor.

• Frequently updated questions
• Objective ground-truth scoring
• Includes instruction following and data analysis categories
• Designed to reduce contamination risk

Stanford CRFM evaluation framework and benchmark collection designed to make model comparisons more transparent across scenarios, metrics, prompts, and adapters.

Useful as a provenance model for the Reliability Floor because it treats prompt, harness, scenario, and metric choices as part of the result rather than invisible background detail.

• Explicit scenario and metric structure
• Records model, adapter, prompt, and metric details
• Covers multiple capability and risk dimensions
• Good reference model for transparent eval reporting

Open benchmark and eval ecosystem focused on robust, reproducible agent testing, with public degradation trackers and historical views for coding agents.

Useful when you want repeated agent and harness measurement rather than a single static benchmark snapshot.

• Degradation Tracker for agent performance drift
• Historical performance views over time
• Benchmark explorers including SWE-Bench Pro and Terminal-Bench 2.0
• Open-source eval runtime that tracks accuracy, tokens, duration, and traces

Ranks models by how often they introduce hallucinations when summarising a source document, scored by the open HHEM evaluation model. Complements SimpleQA by measuring grounded-summarisation faithfulness rather than short-form recall.

Promoted from the KOL-4013 link backlog. Useful second hallucination signal for the Reliability Floor factuality dimension.

• Document-grounded hallucination rate, refreshed as new models land
• Backed by the openly published HHEM scorer
• Reports factual-consistency, answer-rate, and average summary length

Measures the energy consumption and efficiency of serving open LLMs, not just their quality. Useful for the operating-envelope view where cost, latency, and energy matter alongside capability.

Promoted from the KOL-4013 link backlog. Fills the efficiency gap the capability-only benchmarks ignore.

• Energy per request and tokens-per-joule across open models
• Hardware-aware serving measurements
• Complements operating-envelope cost and latency metrics

A collection of prompts that embed misleading or distracting context to test whether a model reasons from first principles or pattern-matches to a wrong but familiar answer. Probes a failure mode standard reasoning benchmarks miss.

Promoted from the KOL-4013 link backlog. Good adversarial-reasoning signal for the reasoning-floor dimension.

• Targets over-reliance on memorised patterns under misleading framing
• Open prompt set with per-model results
• Complements clean-prompt reasoning benchmarks with an adversarial angle

Research organisation tracking the compute, data, cost, and capability trends behind frontier AI. Not a per-model benchmark, but a primary source for the macro trend context the hub cites around model progress.

Promoted from the KOL-4013 link backlog as a cited data source rather than a leaderboard.

• Longitudinal data on training compute, dataset size, and cost
• Independent analysis of AI progress and timelines
• Citable primary source for trend and forecasting context

Artificial Analysis benchmark measuring both how much factual knowledge a model holds and how often it hallucinates when it does not know. Pairs a knowledge-accuracy score with an explicit hallucination rate, so a confident-but-wrong model scores worse than one that abstains.

Promoted from the KOL-4013 top-priority link batch. Strong complement to SimpleQA and the Vectara leaderboard for the reliability/factuality view.

• Separates knowledge breadth from hallucination tendency
• Rewards calibrated abstention over confident guessing
• Refreshed by Artificial Analysis alongside their model coverage