0-100 saturation. Higher is better; 100 means complete.
Dossier / live reference cards
Every card is a fact.
Refreshed 1h benchmarks / meta 10 freshness lanes
Updated
1hRecomputed benchmark-weighted quality scores1hSynced Chatbot Arena benchmark track1hUpdated speed measurements1hValidated official pricing snapshots1hPulled latest OpenRouter price index10hJobs market snapshot refreshed25 MayOpenGuardrails: An Open-Source Context-Aware AI Guardrails Platform25 MayPublished the 2026-05-25 daily digest25 MayWorkbench Launches Open Source BullMQ Dashboard For Node Backends24 MaySpecBench Tests Reward Hacking In Long Horizon Coding Agents1hRecomputed benchmark-weighted quality scores1hSynced Chatbot Arena benchmark track1hUpdated speed measurements1hValidated official pricing snapshots1hPulled latest OpenRouter price index10hJobs market snapshot refreshed25 MayOpenGuardrails: An Open-Source Context-Aware AI Guardrails Platform25 MayPublished the 2026-05-25 daily digest25 MayWorkbench Launches Open Source BullMQ Dashboard For Node Backends24 MaySpecBench Tests Reward Hacking In Long Horizon Coding Agents
AI/RH Monitor
Search models, benchmarks, news Cmd K Hourly source check is green.
FrontierCheapestFastestOpenValue benchmarks / meta - live route
airh:///benchmarks/meta/ [local] [readonly] [live]
AI Resource Hub
v2 / terminal Refreshed 1h benchmarks / meta [?] help - [esc] visual
# route boot: benchmarks / meta
airh > since-last-visit
1h Recomputed benchmark-weighted quality scores
1h Synced Chatbot Arena benchmark track
1h Updated speed measurements
1h Validated official pricing snapshots
airh > open /benchmarks/meta/
AI Resource Hub benchmark draft
Meta Benchmark Hub
Four current views of model reality. 100 = saturated.
10 models 4 leaderboards DSWE metrics Fable pending
Source coverage, not model quality.
Agentic coding plus cost, time, and output tokens.
How much trust to put in the provisional score.
| # | Model | Score | DSWE | Cost | Time | Tokens | Evidence % | |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 Anthropic | 84 ? | Not reported | -- | -- | -- | 74% | |
| 2 | Claude Opus 4.8 Anthropic | 80 ? | 58% / max | $12.58 | 43m | 136k | 82% | |
| 3 | GPT-5.5 xhigh OpenAI | 78 ? | 70% / xhigh | $6.61 | 21m | 47k | 70% | |
| 4 | Gemini 3.1 Pro | 77 ? | Not reported | -- | -- | -- | 76% | |
| 5 | Qwen3.7 Max Alibaba | 74 ? | Not reported | -- | -- | -- | 62% | |
| 6 | Gemini 3.5 Flash | 73 ? | 28% / medium | $7.42 | 17m | 189k | 68% | |
| 7 | MiniMax-M3 MiniMax | 71 ? | 20% | $5.57 | 57m | 98k | 64% | |
| 8 | Kimi K2.6 Moonshot AI | 70 ? | 24% | $3.16 | 56m | 84k | 64% | |
| 9 | GLM-5.1 Reasoning Z.ai | 67 ? | 18% | $7.46 | 35m | 49k | 58% | |
| 10 | DeepSeek V4 Pro DeepSeek | 66 ? | 8% | $4.22 | 37m | 50k | 54% |
Floor Capability: 0-100 saturation score. 100 = complete. Click a model for detail.
| # | Model | Ops score | DSWE | Cost | Time | Tokens | Note |
|---|---|---|---|---|---|---|---|
| 1 | GPT-5.5 xhigh OpenAI | 86 | 70% / xhigh | $6.61 | 21m | 47k | Best tracked DSWE pass rate with relatively low cost, short time, and compact output. |
| 2 | Gemini 3.5 Flash | 78 | 28% / medium | $7.42 | 17m | 189k | Fast DSWE time, but output token volume is very high. |
| 3 | Kimi K2.6 Moonshot AI | 77 | 24% | $3.16 | 56m | 84k | Cheapest tracked DSWE row, but slow and verbose enough to cap the score. |
| 4 | DeepSeek V4 Pro DeepSeek | 72 | 8% | $4.22 | 37m | 50k | Lower DSWE pass rate, but cost, time, and output tokens are contained. |
| 5 | MiniMax-M3 MiniMax | 70 | 20% | $5.57 | 57m | 98k | Moderate DSWE cost but slow average time. |
| 6 | GLM-5.1 Reasoning Z.ai | 68 | 18% | $7.46 | 35m | 49k | DSWE cost and time are mid-pack, with compact output tokens. |
| 7 | Gemini 3.1 Pro | 67 | Not reported | -- | -- | -- | High speed and long context help, but no DSWE row keeps the operating score provisional. |
| 8 | Qwen3.7 Max Alibaba | 62 | Not reported | -- | -- | -- | Pricing is visible, but verbosity and missing DSWE rows keep the operating evidence thin. |
| 9 | Claude Opus 4.8 Anthropic | 60 | 58% / max | $12.58 | 43m | 136k | Strong DSWE pass rate, but high average cost and token burn pull the operating score down. |
| 10 | Claude Fable 5 Anthropic | 58 | Not reported | -- | -- | -- | Fable 5 DSWE is pending, so the operating score is capped until cost, time, and token rows exist. |
Operating Envelope: DSWE-style cost, average time, output tokens, and run friction. External operating rows are retained separately from capability.
| # | Model | Reality score | Primary signal | Floor | DSWE | Evidence % | |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 Anthropic | 92 | AA Index 64.9 (#1) | 84 | Not reported | 74% | |
| 2 | Claude Opus 4.8 Anthropic | 88 | AA Index 61 / AA-Omniscience #2 | 80 | 58% / max | 82% | |
| 3 | GPT-5.5 xhigh OpenAI | 87 | AA Index 60 / Kilo #1 coding | 78 | 70% / xhigh | 70% | |
| 4 | Gemini 3.1 Pro | 86 | AA Index 63 / 148 tok/s | 77 | Not reported | 76% | |
| 5 | Qwen3.7 Max Alibaba | 81 | AA Index 56.6 | 74 | Not reported | 62% | |
| 6 | Gemini 3.5 Flash | 79 | AA Index 55 / Kilo #4 coding | 73 | 28% / medium | 68% | |
| 7 | MiniMax-M3 MiniMax | 77 | AA Index 54.7 / Kilo #10 coding | 71 | 20% | 64% | |
| 8 | Kimi K2.6 Moonshot AI | 76 | AA Index 54 / Kilo #6 coding | 70 | 24% | 64% | |
| 9 | GLM-5.1 Reasoning Z.ai | 72 | AA Index 51 / Kilo #8 coding | 67 | 18% | 58% | |
| 10 | DeepSeek V4 Pro DeepSeek | 71 | AA open top-3 / Index 52 | 66 | 8% | 54% |
Frontier Reality: compact aggregate of AA, Arena, DeepSWE, Kilo, and similar current rows. Public signal first, internal prompts later.
| # | Model | Trust score | Evidence % | Signal | Caveat | |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 Anthropic | 82 draft | 74% | AA Index 64.9 (#1) | Strong external capability signal, but fallback routing keeps the research-trust score provisional. | |
| 2 | Claude Opus 4.8 Anthropic | 82 draft | 82% | AA Index 61 / AA-Omniscience #2 | Best clean Anthropic row in the current set, with hallucination and accuracy evidence available. | |
| 3 | Gemini 3.1 Pro | 76 draft | 76% | AA Index 63 / 148 tok/s | Strong current Gemini intelligence row; needs more direct source-discipline prompt runs. | |
| 4 | GPT-5.5 xhigh OpenAI | 75 draft | 70% | AA Index 60 / Kilo #1 coding | Capability signal is strong, but abstention, citation, and hallucination evidence is thinner. | |
| 5 | Qwen3.7 Max Alibaba | 70 draft | 62% | AA Index 56.6 | Good current Qwen signal, but verbose output raises source-discipline and citation-risk questions. | |
| 6 | Gemini 3.5 Flash | 68 draft | 68% | AA Index 55 / Kilo #4 coding | Useful current Google row; needs internal hallucination and instruction-following runs. | |
| 7 | MiniMax-M3 MiniMax | 67 draft | 64% | AA Index 54.7 / Kilo #10 coding | Promising current/open-weight signal; provider and weight-release variance still need tracking. | |
| 8 | Kimi K2.6 Moonshot AI | 66 draft | 64% | AA Index 54 / Kilo #6 coding | Strong open/current model, but token volume and verbosity make trust scoring cautious. | |
| 9 | GLM-5.1 Reasoning Z.ai | 64 draft | 58% | AA Index 51 / Kilo #8 coding | Current GLM row is useful, but floor-specific honesty and citation data is thin. | |
| 10 | DeepSeek V4 Pro DeepSeek | 62 draft | 54% | AA open top-3 / Index 52 | Current open-model baseline; source-discipline evidence is not yet deep enough. |
Research Trust is draft until fact-checking, bias, fallacy, and deep-research prompt packs are run. Designed to resist quick saturation.
Explainer, method and citations
Score
25% accuracy + grounding / 20% instruction discipline / 15% honesty + reliability / 15% reasoning floor / 15% agent-coding / 10% operating envelope. Floor Capability is the finished benchmark shape. A score of 100 means that lane is saturated.
Watchlist
Fable 5 DSWE score is pending. Mistral top model remains off the public ranking until comparable evidence exists.
Next benchmark lanes
Frontier Reality aggregate, Operating Envelope index, and Research Trust score were the other benchmark directions from the original brief.
Why 100?
100 means routine failures have effectively disappeared under the benchmark lane. If every model reaches 100, that lane is saturated and should be retired or made harder.
Citations
DeepSWE Artificial Analysis AA Intelligence Index Kilo Arena OpenRouter LiveBench HLE Terminal-Bench IFBench SciCode
Benchmark build plan
Comparison gates
Modes are separate score lanes
Plain chat, tool chat, deep research, and agent harnesses can all be useful. They should not be averaged into one brand score.
| Mode | Status | Comparison rule | Evidence captured |
|---|---|---|---|
| Plain chat plain_chat | ready | Compare only against other fresh-chat, no-tool runs. | Prompt text, answer text, model label, plan, run number, scorer notes. |
| Tool-enabled chat tool_enabled_chat | ready | Compare only when the same visible tool class is allowed. | Everything from plain chat plus visible tool calls, citations, files, retries, and tool errors. |
| Deep research deep_research | draft | Compare against other product research modes with source-quality caveats. | Source list, citation accuracy, source class, elapsed time, synthesis quality, and date handling. |
| Agent harness agent_harness | draft | Compare only with harness, filesystem, tool, and permission context attached. | Trace, changed files, commands, browser evidence, blockers, and handoff quality. |
Scoring dimensions
Floor score shape
The score rewards boring correctness, but the caps punish confident unsupported claims, fake tool use, and weak critical dimensions.
| Dimension | Weight | Gate | Saturation risk | Target behaviour |
|---|---|---|---|---|
| Factual accuracy factual_accuracy | 20% | Critical | medium | Correct on boring verifiable facts without drifting into confident guesses. |
| Grounded evidence use grounded_evidence_use | 15% | Critical | low | Keeps claims tied to supplied or verified evidence and exposes missing support. |
| Instruction following instruction_following | 20% | Critical | high | Follows exact constraints, formats, exclusions, and ordering. |
| Honesty and abstention honesty_abstention | 15% | Critical | low | Says when it does not know, separates inference from evidence, and avoids laundering guesses. |
| Reliability and repeatability repeatability | 10% | Supporting | low | Stable enough across 5, 10, and 20 run passes that a single lucky run is not rewarded. |
| Bias and fallacy handling bias_fallacy | 10% | Supporting | medium | Finds argument failures without false balance, partisan projection, or invented motives. |
| Research and tool discipline research_tool_discipline | 10% | Supporting | low | Uses tools only when allowed, records what happened, and does not invent hidden state. |
Benchmark map
Families to build
Each family gets its own prompt pack, answer keys, hard-fail flags, and run records.
| Family | Purpose | Modes | Prompt target | State |
|---|---|---|---|---|
| Floor Capability floor-capability | The headline manual benchmark for everyday trust basics. | Plain chat, Tool-enabled chat | 40 | prompt-pack-needed |
| Fact-Checking fact-checking | Verifies source discipline, contradiction handling, and citation restraint. | Plain chat, Deep research | 20 | prompt-pack-needed |
| Bias And Fallacy bias-fallacy | Tests argument analysis without overreach or false balance. | Plain chat | 20 | prompt-pack-needed |
| Deep Research deep-research | Compares research-mode source discovery, citation accuracy, synthesis, and caveats. | Deep research | 15 | prompt-pack-needed |
| Agent Adaptation agent-adaptation | KOL-3748/FutureSim-style dated information stream testing for belief updates and action routing. | Agent harness | 10 | prompt-pack-needed |
| Creator/Solver Meta-Benchmark creator-solver | BenchBench-style test for whether models can design hard-but-solvable benchmark tasks. | Plain chat, Agent harness | 12 | architecture-ready |
Database-first collection
Raw data tables
The interface is currently fed by a typed workbench module. The table shape below maps to the Postgres-ready schema.
| Table | Purpose | Key fields |
|---|---|---|
| benchmark_scores | Existing AI Resource Hub external benchmark score cache used as one input to the frontier floor meta score. | model_id, benchmark_id, score, source, source_url, measured_at, updated_at |
| meta_benchmark_families | Defines each internal benchmark family and whether it is public, private, draft, or retired. | id, label, purpose, status, created_at, updated_at |
| meta_benchmark_prompts | Stores versioned prompt text, source-pack references, answer keys, rubrics, and holdout state. | id, family_id, version, mode_allowed, difficulty_band, private_holdout, retire_after |
| meta_benchmark_runs | Stores each manual run as model plus product plan plus mode plus date plus run number. | id, prompt_id, provider, product_plan, model_label_shown, tool_mode, run_number, run_datetime |
| meta_benchmark_scores | Stores dimension scores, hard-fail caps, scorer notes, caveats, and final floor score. | run_id, dimension_scores_json, hard_fail_flags_json, score_total, scorer_id, scored_at |
| meta_benchmark_operating_metrics | Stores visible cost, speed, token, tool-call, retry, citation, and elapsed-time metrics. | run_id, wall_time_seconds, visible_tool_calls, visible_citation_count, retry_count, output_length_words |
| external_benchmark_operating_metrics | Stores DSWE-style source-reported benchmark score, average cost, average time, and output token rows for external benchmarks. | benchmark_id, source_name, model_id, run_configuration, score_value, average_cost_usd, average_time_seconds, output_tokens_average |
| meta_benchmark_sources | Stores source packs and source evidence used by fact-checking, deep research, and adaptation tasks. | id, source_type, url, source_pack_id, verified_at, caveat |
Next data collection
First pilot queue
No model result is shown here until it has prompt version, mode, run number, scorer notes, and hard-fail state.
| Pilot | Family | Mode | Next action | Data captured | Status |
|---|---|---|---|---|---|
| pilot-floor-001 | Floor Capability | Plain chat | Build first 8 to 12 prompts and answer keys before any model run. | Prompt, answer, score dimensions, hard fails, repeat-run variance. | ready-to-build |
| pilot-fact-001 | Fact-Checking | Plain chat | Create public source packs with supported, contradicted, and unsupported claims. | Claim labels, source refs, quote discipline, fabricated citation flags. | needs-source-pack |
| pilot-research-001 | Deep Research | Deep research | Define 5 public research tasks with expected source classes and citation checks. | Source list, citation accuracy, elapsed time, caveats, synthesis score. | not-started |
| pilot-agent-001 | Agent Adaptation | Agent harness | Turn KOL-3748 into 10 dated information-stream tasks. | Search timing, belief update, memory preservation, action routing, evidence quality. | ready-to-build |
airh > open /benchmarks/meta/ Run
top 10compare gpt claudenews --todayprice sonnetvisual