What's new

Transparency

Every source and every formula

This page lists where each number on the site comes from, how it gets here, whether the source's terms allow it, and the sum behind anything we work out ourselves. It also lists what is wrong today. Some of our data is typed in by hand, some is out of date, and two sources have been removed because their terms restrict our use.

Last reviewed against the code and the sources: 14 November 2026. Data on the site was last rebuilt: 19 Nov 2026, 14:58 UTC.

AI and jobs: figure-level receipts

Indeed Hiring Lab's AI Tracker, CC BY 4.0. The pinned repository README and licence were read first-hand on 14 November 2026. Daily observations use a seven-day trailing average; source refresh is monthly. Only Indeed postings are covered; these are keyword mentions, not employment effects. No uncertainty published by the source. Every displayed percentage is selected directly from the pinned CSV and rounded for display; the export retains original precision.

Retrieved 2026-11-14 · Immutable snapshot, method versions, source commit and raw SHA-256

All published observations: evidence kind, date, method and rights
Country / kindPeriod / valueVintage / uncertaintySource / method / rights
AU · Measured2025-09-30 · 4.86634876733584%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
AU · Measured2026-09-30 · 7.12029880911784%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
CA · Measured2025-09-30 · 5.74057477886402%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
CA · Measured2026-09-30 · 18.9696563904776%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
DE · Measured2025-09-30 · 3.59807946826636%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
DE · Measured2026-09-30 · 5.74163012331981%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
FR · Measured2025-09-30 · 2.75868279459262%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
FR · Measured2026-09-30 · 4.52739491947547%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
GB · Measured2025-09-30 · 5.1227875501628%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
GB · Measured2026-09-30 · 9.92103324690105%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
IE · Measured2025-09-30 · 9.85378158454038%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
IE · Measured2026-09-30 · 15.0197438100594%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
IT · Measured2025-09-30 · 3.25824189284218%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
IT · Measured2026-09-30 · 4.76487285784418%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
NL · Measured2025-09-30 · 2.3265266903427%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
NL · Measured2026-09-30 · 3.62811746725153%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
US · Measured2025-09-30 · 3.70551075376619%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0
US · Measured2026-09-30 · 7.33818434887748%2026-10-08 · No uncertainty published by the source · Within refresh windowCSV · Method version · CC BY 4.0

Credit: Source: Indeed Hiring Lab. Licensed under CC BY 4.0. Selected observations; formatting adapted by AIRH. Licence: CC BY 4.0. Terms read on 2026-11-14.

Rights remain unconfirmed for unused sources until the specific dataset's terms are read. No ONS, BLS, Anthropic, Eurostat, OECD, ILO or company-claim figures are copied into this section. Reviewed source policy and data model.

Lab Standings: every input and calculation

Method v0.3. Default weights: 35, 20, 15, 10, 20 for capability, value, shipping pace, openness and developer usage. These are editorial choices, not measured facts. Capability is progress above the same-download frontier a year earlier; value uses log(1 + progress / blended price), normalised to the leader. Blend = (3 × input + output) / 4, an explicit workload assumption. Shipping and usage use square-root ratios; openness averages checked licence credits. Missing inputs never become zeroes.

Full formulas, coverage threshold, sensitivity and terms. Epoch's included benchmarks and licence read on 14 November 2026: no Artificial Analysis or Arena sources listed in the fit. The published index may be revised by Epoch; AIRH preserves each downloaded vintage.

2026-11-14

All original inputs, source URLs, dates, SHA-256 version, licences and derived scores

Unmapped models are retained in the audit, excluded from scores. Shipping is withheld without a release-channel check. Open weights are withheld without a checked licence. The bench in PR #111 supplies synthetic fixture evidence only; no model comparison is inferred.

4

Collected automatically

4

Entered by hand

1

Out of date

2

Removed from the site

1

Not collected yet

What is wrong today

We found these while writing this page. They are listed first because a reader deserves to see them before the tidy tables.

  • Speed figures have been removed. They came from a fixed list recorded between 15 January and 20 February 2026 that was re-saved on every run, and the source's terms restrict our use. We have no licensed replacement yet.
  • Most benchmark scores have no source link. 324 of 365 were entered without one. Those are held back from rankings until each has a link to where the score was published.
  • Hand-entered prices can look fresher than they are. The date beside a lab's list price is the time of our last run, not the last time someone checked the lab's page.
  • Release dates are often listing dates. For models added through OpenRouter, the date is the day OpenRouter listed the model. That is usually within days of the launch, but not always.
  • Four company job feeds return nothing. Google DeepMind, Mistral, Runway and Wayve are missing from the jobs data.
  • Price history mixes sources. A model's history can jump between a lab's list price and a reseller's cheapest price. We do not publish statistics on how often prices change until that is fixed.

Sources, terms and refresh times

"Last run" is the last time our collector ran, read from its log when this page was built. A run does not prove the figure changed, and for hand-entered data it does not prove anyone re-checked the source.

OpenRouter models API

Collected automaticallyPublic API or feed
What we take
Input and output price per million tokens, context window, maximum output, modality and the date the model was first listed.
How
One request to the public API each run. Prices arrive per token and are multiplied by one million, then rounded to three decimal places. Free, batch and other routing variants are skipped.
Terms
Terms review pending. It is a public API that needs no key, and we found no data licence covering it when we read OpenRouter's Terms of Service (last updated 31 August 2026) on 14 November 2026. Those terms bar scraping the website by automated means; we use only the API.
Refresh
Hourly · last run 19 Nov 2026, 14:58 UTC
Shown on
Model pages, Compare, Cheapest, Context window, Pricing trends
Caveat
The price is the cheapest host OpenRouter lists for that model, which can be lower than the lab's own list price. For these models the "released" date is the day OpenRouter listed the model, not the lab's announcement date. About three quarters of the language models in the catalogue get their price this way.

Labs' own pricing pages

Entered by handPublished facts, quoted with a link
What we take
List prices for input and output tokens for the models each lab sells directly.
How
Typed in by hand from each lab's pricing page into our model catalogue. Nothing is scraped from these pages. For a model in the catalogue, its pages show this list price rather than the OpenRouter price; the stored OpenRouter price is kept, not overwritten.
Terms
Published list prices, quoted with a link to the lab's page.
Refresh
When someone re-checks the page · last run 19 Nov 2026, 14:58 UTC
Shown on
Model pages, Compare, Calculator
Caveat
The "price updated" date on these rows is the date a person last checked the lab's page, where that date is recorded; otherwise no date is shown. Earlier versions showed the time of the last run instead, which made old prices look fresh. That fault is fixed, and run-time dates recorded before the fix are not evidence of a check.

Our model catalogue

Entered by handPublished facts, quoted with a link
What we take
Which models and providers exist, their names, release dates, lifecycle status and links to official documentation.
How
Maintained by hand from official announcements and documentation. New paid models from watched providers are also added automatically when OpenRouter lists them, with the status "tracking".
Terms
Our own compilation.
Refresh
Hourly sync of a hand-maintained file · last run 19 Nov 2026, 14:58 UTC
Shown on
Models, Labs, New releases
Caveat
Coverage follows what we have added and what OpenRouter lists. It is strongest for the large US labs and thinner elsewhere. The open-source flag is unreliable for automatically added models, so we do not publish open-versus-closed statistics yet.

Benchmark scores entered by hand

Entered by handPublished facts, quoted with a link
What we take
Scores on public benchmarks, mostly as reported in lab announcements, model cards and papers. The set also holds older arena ratings, which are covered under Arena below.
How
Typed in by hand. The set is rebuilt from the same file on every run.
Terms
Figures quoted from the publishing lab. Each should link to where it was published.
Refresh
When someone adds or corrects a score
Shown on
Benchmarks, Leaderboard, Model pages
Caveat
Of 365 scores, 324 have no source link recorded yet. Scores without a traceable source are held back from rankings until the link is added. A lab's own score is a claim by that lab, not an independent test.

Artificial Analysis

Removed from the siteTerms restrict our use
What we take
Output speed (tokens per second), time to first token, and arena ratings for image, video and voice models.
How
No longer collected. Its figures are dropped before any page, export or score is built.
Terms
Data Platform Terms (revised 19 Aug 2026), section 2.5, bar use of its data in ranking or comparison products without written consent, and 2.4 bars redistribution; its website Terms of Use (revised 15 Sep 2026) bar automated requests to the site (3.3(b)(vi)) and building a competitive site (2.2(c)). Terms checked on 14 November 2026. Read the terms.
Refresh
Not collected
Shown on
Speed, Fastest, Creative model ratings
Caveat
Speed and creative arena figures from this source have been removed from the site. No article on this site is built on them.

Arena (formerly LMArena and LMSYS Chatbot Arena)

Removed from the siteTerms restrict our use
What we take
Arena Elo ratings for language models.
How
No longer collected. Its ratings are dropped before any page, export or score is built.
Terms
Terms of Use (updated 23 Feb 2026) bar programmatic access, manual or automated extraction of data, and reproduction or commercial use of the service output. Terms checked on 14 November 2026. Read the terms.
Refresh
Not collected
Shown on
Benchmarks, Leaderboard
Caveat
Arena ratings have been removed from the site and from our composite scores. No article on this site is built on them.

Company job boards (Greenhouse and Lever)

Collected automaticallyReview pending
What we take
Open roles at AI companies: title, team, location, workplace type and posting date.
How
The public job-board API each company uses for its own careers page.
Terms
Public job-board APIs built for listing a company's roles. A written terms review is still to do.
Refresh
Hourly · last run 19 Nov 2026, 14:58 UTC
Shown on
Jobs
Caveat
Six companies are collected today. The feeds for Google DeepMind, Mistral, Runway and Wayve are returning nothing, so their roles are missing. We do not publish hiring statistics until those are fixed and we have more than one snapshot.

Provider status pages

Collected automaticallyPublic API or feed
What we take
Current service status and recent incidents for the main model providers.
How
Each provider's public status feed (Statuspage JSON or RSS).
Terms
Public status feeds published for this purpose.
Refresh
Hourly
Shown on
Status

News feeds

Collected automaticallyPublic API or feed
What we take
Headlines, links and short summaries from lab blogs, research feeds and technology news sites.
How
Two routes. The collector in this repository fetches seven feeds (OpenAI, Anthropic, Google, TechCrunch, VentureBeat, The Verge and Ars Technica). A shared news pipeline, used by several of our sites, is configured with 14 sources and supplies the rest. We link to the original and never republish full articles. Lab sources are treated as primary; media sources are cross-checked before a story is promoted.
Terms
Feeds published for syndication. Headline, link and a short summary only.
Refresh
Hourly · last run 19 Nov 2026, 14:58 UTC
Shown on
News, Daily digest, Newsletter
Caveat
The last-run time shown here is for the seven-feed collector only. It says nothing about the health of the shared pipeline. Some feeds rate-limit us from time to time, so a source can be missing from a given run.

Consumer subscription plans

Out of datePublished facts, quoted with a link
What we take
Monthly prices and usage limits for consumer AI plans.
How
Typed in by hand from each provider's pricing page.
Terms
Published list prices.
Refresh
When someone re-checks the page
Shown on
Compare plans
Caveat
These rows have not been re-checked recently and at least one source link points at the wrong page. Do not rely on them until this entry changes.

AI events calendar

Entered by handPublished facts, quoted with a link
What we take
Names, dates and locations of major AI conferences.
How
Entered by hand from each event's official site. An automatic check confirms each event page still loads.
Terms
Facts from official event pages, with links.
Refresh
When events are announced
Shown on
Events

Epoch AI

Not collected yetOpen licence
What we take
Nothing yet.
How
We link to Epoch's research and plan to use its capability and company datasets, which are published under CC BY 4.0. No Epoch figures are on the site today.
Terms
CC BY 4.0, with attribution to epoch.ai/data.
Refresh
Not collected

Practitioner guides: sources and worked examples

These guides propose working methods. Example workloads and outcomes are explicitly assumed, not model measurements. Reproduce their frozen tables with node scripts/insights/practitioner-examples.mjs --check and node scripts/insights/practitioner-operations.mjs --check. Documentation is linked and briefly paraphrased, not collected as a dataset; no additional data licence is claimed.

Upgrade and price-provenance examples: node scripts/insights/practitioner-tracking.mjs --check.

Choose an AI model by the job you need finished

Illustrative request cost = (input tokens × $1 + output tokens × $5) / 1,000,000. Both rates are assumptions, not a quote for a named model.

Inputs: Four invented workloads use assumed rates of $1 per million input tokens and $5 per million output tokens. No model was tested.

Limits: These sums omit caching, tools, reasoning, retries, taxes and review time. They do not establish which model can do any of the jobs.

  • Claude context windows, read 14 November 2026: The input, tool definitions and generated output share context capacity.

Code: scripts/insights/practitioner-examples.mjs. Frozen tables: src/data/insights/2026-11-14-practitioner.json.

The useful AI price is the cost per accepted answer

Cost per 1,000 accepted results = total measured API spend / accepted results × 1,000. Acceptance rate = accepted results / attempts. A zero accepted count has no finite cost per accepted result.

Inputs: Three fictional candidates each make 1,000 attempts. Spend and accepted counts are invented to explain the arithmetic, not observations.

Limits: Acceptance depends on a task-specific rule. Aggregate spend includes failed attempts in this illustration; human time and downstream losses are excluded.

  • Claude structured outputs, read 14 November 2026: Schema constraints govern output format; refusals and token-limit stops need separate handling.

Code: scripts/insights/practitioner-examples.mjs. Frozen tables: src/data/insights/2026-11-14-practitioner.json.

Prompt caching pays only when the prefix gets used again

For N uses of one unchanged prefix: ordinary cost = N base units; cached cost = write multiplier + (N − 1) × read multiplier. Cache is cheaper when N > (write − read) / (1 − read). Examples use write 1.25 or 2 and read 0.1.

Inputs: One base unit is the ordinary input cost of the repeated prefix. One successful write is followed by N − 1 hits before expiry. New input and output are excluded.

Limits: These are standard Claude multipliers read on the stated date, not universal rates. Selected models have lower read multipliers. Minimum prefix lengths, changes and expiry can prevent hits.

  • Claude prompt caching, read 14 November 2026: Standard five-minute writes cost 1.25 times base input, one-hour writes 2 times, and reads 0.1 times, with model exceptions. Reuse requires a matching prefix; hits refresh the lifetime.

Code: scripts/insights/practitioner-examples.mjs. Frozen tables: src/data/insights/2026-11-14-practitioner.json.

The batch discount needs a deadline you can afford

Illustrative completed-job bill = (1,000 − expired requests) × $0.01 + expired requests × $0.02. Saving = 1 − bill / $20. Expired requests are assumed unbilled and each succeeds on one ordinary-price fallback.

Inputs: 1,000 independent jobs cost an assumed $0.02 each ordinarily, or $0.01 in a batch. Scenarios have 0, 100 or 300 expiries, each followed by one successful ordinary request.

Limits: Completion and expiry counts are invented. This is a narrow cost scenario, not a completion forecast. Errors that were billed, retries, review and orchestration costs would change it.

  • Claude batch processing, read 14 November 2026: Message Batches offer a 50% discount, expire after 24 hours, use custom_id for matching unordered results, and do not charge expired requests. Cache hits in batches are best effort.

Code: scripts/insights/practitioner-operations.mjs. Frozen tables: src/data/insights/2026-11-14-operations.json.

A long document can cost you again on every turn

With a 20,000-token initial prefix and 1,000 extra input tokens per later turn, cumulative replay input over N turns = 20,000 × N + 1,000 × N × (N − 1) / 2. Input cost = tokens / 1,000,000 × an assumed $1 rate. Fixed-prefix baseline = 20,000 × N.

Inputs: A synthetic application resends its full prefix and retained history on each request. History grows by exactly 1,000 tokens between requests. No caching, trimming, compaction or output charges.

Limits: This describes the assumed application, not every chat product. Real tokenisation, reasoning retention and context-management behaviour differ. The fixed-prefix column is a mathematical baseline, not a quality-equivalent replacement.

  • Claude context windows, read 14 November 2026: Conversation history occupies context and generated responses can become later input. Cached prefixes still occupy the window.

Code: scripts/insights/practitioner-operations.mjs. Frozen tables: src/data/insights/2026-11-14-operations.json.

Valid JSON is only the first pass for an AI extractor

Each funnel percentage = count passing the named cumulative gate / 1,000 original requests × 100. Task acceptance among schema-valid results = 900 / 970 × 100, about 92.8%.

Inputs: A made-up run starts with 1,000 completed HTTP requests, then has 980 usable structured completions, 970 schema-valid outputs and 900 task-valid results. Counts are nested, not independent.

Limits: No extraction test was run. These are proposed reporting gates. Task validity needs reviewed evidence; it cannot be inferred from parsing or a schema guarantee.

  • Claude structured outputs, read 14 November 2026: JSON outputs constrain response shape. Refusals can return HTTP 200 without matching the schema, and max_tokens can leave incomplete output. Supported schemas have limits.

Code: scripts/insights/practitioner-operations.mjs. Frozen tables: src/data/insights/2026-11-14-operations.json.

A model upgrade needs a regression register

Old passes = both-pass + old-only = 88 + 7 = 95. New passes = both-pass + new-only = 88 + 4 = 92. On 100 cases, net change = −3 percentage points; regressions = 7 and gains = 4.

Inputs: A fictional paired test has 100 fixed cases, each run once on an old and a new candidate under one acceptance rule. The counts are invented.

Limits: One run is not a stability estimate and the case mix is not a population sample. A provider replacement recommendation is not proof of equal behaviour on a particular workflow.

  • Claude model deprecations, read 14 November 2026: Deprecated and retired are distinct states; retired requests fail. Partner-operated platforms may have different retirement schedules. The guide recommends testing replacement models before retirement.

Code: scripts/insights/practitioner-tracking.mjs. Frozen tables: src/data/insights/2026-11-14-tracking.json.

A cheaper tracker row may be a different seller

Apparent price change = (later output rate − earlier output rate) / earlier output rate × 100 = ($1 − $2) / $2 × 100 = −50%. Because the seller changes, this is not evidence of a same-seller price cut.

Inputs: Two fictional output-price records quote US dollars per million tokens. The earlier record is Host A at $2 and the later Host B at $1. No market trend or real price cut is claimed.

Limits: The example illustrates source continuity only. A valid same-seller comparison also needs the same model version, billing unit, service tier and applicable conditions. No Artificial Analysis or Arena figures are used.

  • OpenRouter provider routing, read 14 November 2026: Routing can select providers and endpoints and configure fallbacks. A model identifier alone does not specify one seller or endpoint.

Code: scripts/insights/practitioner-tracking.mjs. Frozen tables: src/data/insights/2026-11-14-tracking.json.

More practitioner guides: evidence and calculations

These examples use stated assumptions. They are not model measurements. Each entry names its frozen data and an offline script; run the script with --check to reproduce it. Documentation is linked and briefly paraphrased, not collected as a dataset. No additional data licence is claimed.

Find out whether your RAG system found the answer first

Retrieval coverage = 80 / 100 = 80%. Accepted answers overall = 68 / 100 = 68%. Accepted answers among retrieved cases = 68 / 80 = 85%. Counts are nested assumptions.

Assumptions: 100 fictional answerable questions: evidence is retrieved for 80 and an accepted answer is produced for 68 of those. No system was evaluated.

Limits: This defines evidence availability for one question set, not document recall across a whole corpus. A retrieved passage can still be insufficient or misread.

  • Anthropic: Contextual Retrieval, read 14 November 2026: The described retrieval workflow chunks documents, retrieves relevant passages and adds them to the generation prompt. Its hybrid example combines lexical and embedding retrieval.

Code: scripts/insights/practitioner-b4.mjs. Frozen tables: src/data/insights/2026-11-14-b4.json.

Overlapping document chunks has a bill you can count

For document length L, chunk capacity C and overlap O, greedy full coverage uses 1 + ceil(max(0,L − C)/(C − O)) chunks. Stored tokens = L + (chunks − 1) × O. Example L=10,000, C=1,000, O=0,200,500.

Assumptions: One synthetic 10,000-token document is split into chunks of at most 1,000 tokens. Each adjacent pair overlaps by a fixed amount; the final chunk may be shorter.

Limits: Token counts are arithmetic, not tokenizer measurements. This ignores metadata and vector storage and does not demonstrate a retrieval-quality benefit.

  • Anthropic: Contextual Retrieval, read 14 November 2026: The described retrieval workflow chunks documents, retrieves relevant passages and adds them to the generation prompt. Its hybrid example combines lexical and embedding retrieval.

Code: scripts/insights/practitioner-b4.mjs. Frozen tables: src/data/insights/2026-11-14-b4.json.

An embedding upgrade is an index migration

On 50 fictional queries: both retrieve evidence=39, old-only=6, new-only=3, neither=2. Old coverage=(39+6)/50=90%; new=(39+3)/50=84%. Migration text volume=100,000 chunks × 600 assumed tokens=60,000,000 tokens.

Assumptions: Two invented retrieval configurations share 50 questions. A separate planning example has 100,000 chunks averaging exactly 600 tokens. No embeddings were generated.

Limits: Equal vector dimensions do not establish model compatibility. The example supplies no real accuracy, rebuild duration, storage forecast or embedding price.

  • Google: embedding task types, read 14 November 2026: Retrieval queries and retrieval documents have distinct task types. Embedding configuration is part of the retrieval setup.

Code: scripts/insights/practitioner-b4.mjs. Frozen tables: src/data/insights/2026-11-14-b4.json.

The first token is not the finished answer

Idealised completion seconds = start delay + 800 output tokens / constant emission rate. A: 0.2+800/40=20.2 seconds; B: 1+800/100=9 seconds. This is a fluid approximation, not an observed stream.

Assumptions: Two fictional routes emit 800 tokens at constant rates after assumed start delays. No queue variation, thinking, tools, network jitter or rendering delay.

Limits: Token emission is not truly continuous. Real streams have varied block types and gaps; a first event need not contain useful answer text. No current speed ranking is made.

  • Claude streaming messages, read 14 November 2026: Messages can stream incrementally as server-sent events. The event flow includes content blocks and a final message_stop; receiving initial content is not the same as a complete message.

Code: scripts/insights/practitioner-b5.mjs. Frozen tables: src/data/insights/2026-11-14-b5.json.

Your request limit may not be the limit you hit

Illustrative steady throughput ceiling = min(RPM, input-token limit / input per request, output-token limit / output per request). With limits 60,120,000,30,000: workloads (1,000,200), (4,000,1,000), (4,000,2,000) give 60,30,15 requests/minute.

Assumptions: Hypothetical limits of 60 requests, 120,000 input tokens and 30,000 output tokens per minute. Fixed per-request sizes, no caching or bursts.

Limits: A ceiling is not a delivered throughput guarantee. Fractional capacity, shared limits, estimation rules, burst controls and other workloads can reduce useful throughput.

  • Claude API rate limits, read 14 November 2026: Messages API rate limits include requests, input tokens and output tokens per minute. Short bursts can hit limits; published maxima are not guaranteed minimum throughput.

Code: scripts/insights/practitioner-b5.mjs. Frozen tables: src/data/insights/2026-11-14-b5.json.

Retries belong in the AI budget before the first request

Illustrative attempts = 100+20+5=125, costing 125×$0.01=$1.25. Accepted jobs =80+15+2=97. A maximum of three attempts per 100 jobs reserves up to 300×$0.01=$3 under the fixed-price assumption.

Assumptions: Fictional job ledger: 100 first attempts yield 80 accepted; 20 second attempts yield 15 more; five third attempts yield two more. Each attempt costs exactly $0.01, even when unsuccessful.

Limits: Real failure probabilities and bills differ. No independence or success forecast is assumed. Three attempts means the first plus at most two retries; transport failure does not by itself prove whether a side effect happened.

  • Claude API errors, read 14 November 2026: The API distinguishes validation, rate-limit, internal and overload errors. Its guidance recommends exponential backoff for internal API errors; error type and context matter when choosing recovery.

Code: scripts/insights/practitioner-b5.mjs. Frozen tables: src/data/insights/2026-11-14-b5.json.

An agent budget needs a stop the application enforces

Assumed maximum model-call charge = (4,000×$1 + 1,000×$5)/1,000,000=$0.009. For caps of 5,10,20 calls, nominal reserves are $0.045,$0.090,$0.180. All billable output must fit the 1,000-token cap.

Assumptions: Hypothetical flat rates and enforceable per-call maxima of 4,000 input and 1,000 billable output tokens. Calls are serial, with no retries outside the ledger.

Limits: This is a model-call bound under fixed assumptions, not a whole-agent spending guarantee. Tools, storage, price tiers, hidden retries and unaccounted parallel calls require separate limits.

  • Claude tool-use overview, read 14 November 2026: For client tools, the model emits structured tool calls and the application executes operations and returns results. The application therefore controls that execution boundary.

Code: scripts/insights/practitioner-b6.mjs. Frozen tables: src/data/insights/2026-11-14-b6.json.

A tool result can use more context than the question

Synthetic tool-result allowance =32,000 total −4,000 fixed input −8,000 reserved output=20,000 tokens. Ten results of 500,2,000,8,000 tokens occupy 5,000,20,000,80,000 tokens respectively.

Assumptions: One hypothetical 32,000-token request budget with fixed input and output reserves. Ten tool results are included once each, with no other history or overhead.

Limits: This is a text-capacity calculation, not a measured tokenizer result or a model specification. Repeated history and tool schemas need additional accounting. Shortening a result can remove needed evidence.

  • Claude: handle tool calls, read 14 November 2026: Client tool results are returned to the model in tool_result blocks associated with tool calls. The returned content is part of the material the application supplies.

Code: scripts/insights/practitioner-b6.mjs. Frozen tables: src/data/insights/2026-11-14-b6.json.

Test what an agent can do after a prompt injection

Fictional screening flags 18/20 malicious fixtures=90%; two reach the model. Of two prohibited action proposals, an independent executor denies 2/2. Executed prohibited actions=0 in this constructed example.

Assumptions: Twenty invented malicious tool-result fixtures, eighteen detected, two passed through, two prohibited proposals and two executor denials. No attack or security assessment was run.

Limits: A constructed zero is not evidence of safety or a real attack success rate. Other permissions, data exposure and benign-task failures require separate tests. No defence is claimed complete.

  • Claude: mitigate prompt injections, read 14 November 2026: Indirect injections can arrive through third-party documents and tool results. The guidance recommends separating untrusted content and applying least privilege, including narrowly scoped permissions and sandboxing.

Code: scripts/insights/practitioner-b6.mjs. Frozen tables: src/data/insights/2026-11-14-b6.json.

A citation can point to a real passage and still fail the claim

Synthetic audit: 100 claims, 95 valid pointers, 80 supported claims among those. Pointer validity=95%; support overall=80%; support among valid pointers=80/95×100≈84.2%. Valid but unsupported=95−80=15.

Assumptions: 100 invented claim-citation pairs, one citation per claim. Ninety-five point to the intended source; eighty of those support the complete claim. No real citation system was tested.

Limits: Valid pointers do not establish source truth, currentness or full claim support. Multiple-source claims need a separate completeness rule. This is not a provider citation-accuracy result.

  • Claude citations, read 14 November 2026: The API returns source-location pointers and cited text. PDF citations use page locations; plain-text citations use character locations. These identify source material for verification.

Code: scripts/insights/practitioner-b7.mjs. Frozen tables: src/data/insights/2026-11-14-b7.json.

High OCR field accuracy can hide a pile of broken documents

100 synthetic documents ×10 fields=1,000 fields. Twenty errors give 98% field accuracy in both scenarios. Concentrated in two documents:98/100 clean=98%. Spread over twenty documents:80/100 clean=80%.

Assumptions: Each document has exactly ten equally counted fields. A document passes only when all ten fields are correct. Twenty wrong fields are arranged in either two or twenty documents.

Limits: The scenarios are constructed and do not describe OCR performance. Real fields differ in importance; repeated values and dependent errors violate any simple independent-error interpretation.

  • Google Document AI: handle a processing response, read 14 November 2026: Document AI responses provide extracted document structure and text anchors that connect structured output to document text. These fields can be inspected when validating extracted results.

Code: scripts/insights/practitioner-b7.mjs. Frozen tables: src/data/insights/2026-11-14-b7.json.

Budget translated prompts from their token counts

Synthetic monthly input =10,000 requests × assumed input tokens. At an assumed $1 per million input tokens, lengths 4,000,7,000,12,000 cost $40,$70,$120. No token counts were measured and no language ratio is claimed.

Assumptions: Three invented localised prompt versions have specified input lengths and the same 10,000-request volume. Flat input-only teaching rate of $1 per million tokens.

Limits: Versions are not named languages and are not claimed semantically equivalent by test. Output, caching, tools, token-estimation differences and quality checks are excluded.

  • Claude token counting, read 14 November 2026: The token-counting endpoint accepts structured message input, including system prompts and supported tool/document inputs. Counting is an estimate; actual message usage can differ.

Code: scripts/insights/practitioner-b7.mjs. Frozen tables: src/data/insights/2026-11-14-b7.json.

Zero failures in a small AI test does not mean zero risk

Under independent identical trials with fixed failure probability p, probability of zero failures in n trials=(1−p)^n. At p=0.01 and n=50,100,300, this is about 60.5%,36.6%,4.9%. For zero observed failures, one-sided exact 95% binomial upper bound=1−0.05^(1/n).

Assumptions: A mathematical model of independent identical trials with a fixed 1% failure probability. The displayed upper bounds assume zero observed failures. No AI system was tested.

Limits: Related fixtures, changing prompts or workload drift can invalidate the trial model. A frequentist bound is not a 95% probability statement about a particular model being safe; it covers only the defined failure and sampling assumptions.

  • NIST: binomial distribution, read 14 November 2026: The binomial model describes counts of binary outcomes over trials with a fixed probability. The zero-failure probability here follows by substituting zero into its probability formula.

Code: scripts/insights/practitioner-b8.mjs. Frozen tables: src/data/insights/2026-11-14-b8.json.

A 99% classifier can miss every case you care about

For 1,000 synthetic cases with ten positives: always-negative gives TP=0,FN=10,FP=0,TN=990, accuracy=99%,recall=0%,precision undefined. Candidate gives TP=8,FN=2,FP=20,TN=970, accuracy=97.8%,recall=80%,precision=8/28≈28.6%.

Assumptions: An invented binary classification task contains ten relevant positive cases and 990 negatives. Two constructed prediction sets are compared at fixed decisions.

Limits: The class mix and errors are synthetic. Precision and recall will depend on the real workload and decision threshold. No model ranking or universal trade-off is established.

  • Google: accuracy, precision and recall, read 14 November 2026: Accuracy counts all correct classifications; recall counts detected actual positives; precision counts correct predicted positives. The guide explains why overall accuracy can mislead on imbalanced data.

Code: scripts/insights/practitioner-b8.mjs. Frozen tables: src/data/insights/2026-11-14-b8.json.

Your AI judge needs a disagreement log

Synthetic 100-case table: both accept=80,reference accept/judge reject=10,reference reject/judge accept=5,both reject=5. Agreement=(80+5)/100=85%; disagreement=15%; false accepts among ten reference rejects=5/10=50%.

Assumptions: One hundred invented answers have binary reviewed reference labels and one automated judge label each. The human reference is the chosen test standard, not infallible truth.

Limits: No judge was run. Agreement is specific to the rubric, case mix and labels. A high overall rate can conceal poor agreement on a small important group; this example does not estimate real judging reliability.

Code: scripts/insights/practitioner-b8.mjs. Frozen tables: src/data/insights/2026-11-14-b8.json.

Figures in our articles

The catalogue-based analyses below use one script, run against a dated copy of our catalogue. The output is saved with the article, so the figures in a piece do not drift after it is published. Anyone can run the script and get the same answers: node scripts/insights/compute-insights.mjs.

Median context window of new models, by half-year

The middle context window among language models first listed in each six-month period.

Take every language model with a release date and a context window. Remove variants. Group by half-year of release (H1 is January to June, H2 is July to December). Report the median of each group. Growth multiple = median of the latest group ÷ median of the earliest group with at least 10 models.

Inputs
Context window and release date from the model catalogue (mostly the OpenRouter models API).
Limits
For most models the release date is the day OpenRouter listed it. Groups with fewer than 10 models are left out of the headline. The catalogue is not every model ever released, so this describes the models we track.
Code
scripts/insights/compute-insights.mjs

Removing variants

Leaves out catalogue entries whose names mark them as a dated snapshot or a fast tier, so that the same model is less likely to be counted more than once.

A catalogue entry is a variant if its id ends in a date stamp (-MMDD, -YYMM, -YYYYMMDD or -YYYY-MM-DD) or in -fast. Variants are dropped before any count or median.

Inputs
Catalogue ids.
Limits
A simple naming rule, not a check that another entry for the same model exists: it can drop a "-fast" entry that has no standard sibling in the catalogue. It removes 23 of 291 language-model entries in the 14 November 2026 snapshot. Point releases with their own name (for example 5.4 and 5.5) still count separately.
Code
scripts/insights/compute-insights.mjs

Price spread

How many times dearer the most expensive model is than the cheapest, per million output tokens.

Highest output price ÷ lowest output price, across language models with the status "active" and a price above zero, variants removed.

Inputs
Output price per million tokens from the model catalogue.
Limits
The cheap end is the cheapest host on OpenRouter, not a lab's list price. The dear end is a lab's list price. Free models are excluded. Both ends were re-checked against OpenRouter's live API on the snapshot date.
Code
scripts/insights/compute-insights.mjs

Output-to-input price ratio

How many times more a model charges for a token it writes than for a token it reads.

Output price ÷ input price for each model, then the median of those ratios for models first listed in each half-year, variants removed.

Inputs
Input and output price per million tokens from the model catalogue.
Limits
Uses today's prices grouped by release period, not the price on launch day. A few rows have extreme ratios that look like data errors; the median is not moved by them.
Code
scripts/insights/compute-insights.mjs

Price-history repeats

How many price-history rows for lab-priced language models are the previous price written down again.

Take language models only, and rows whose source is not OpenRouter. For each model, order the rows by time and leave out the first. A later row repeats when its input and output prices both equal the previous row of the same model. Rows labelled with a lab name and "(official)" and rows labelled with a website address are also counted separately. Repeat share = repeats ÷ rows counted.

Inputs
Rows of price_history.json and models.json in the committed cache; the script records the SHA-256 of both.
Limits
Describes our records, not how often labs change prices. A change between two rows can be a real move, a correction or a change of source label; the history cannot tell them apart.
Code
scripts/insights/price-history-repeats.mjs

Figures on the site

These are the sums behind the scores and percentages on the reference pages. The methodology page explains the reasoning; this section gives the arithmetic.

Blended price

Used on: Best value, Cheapest, model pages.

A single price per million tokens that assumes a typical job reads one part and writes three.

(input price + 3 × output price) ÷ 4

Limits
The 1:3 mix is a convention. Chat with long documents reads far more than it writes, so the blended price overstates the cost of that kind of work.
Code
src/data/hub-dashboard.ts

Price change

Used on: Model pages, Pricing trends.

The percentage move between two consecutive price records for one model.

(new blended price − previous blended price) ÷ previous blended price × 100

Limits
The history mixes sources. A switch from a lab's list price to OpenRouter's cheapest host shows up as a price move when nothing changed. We do not publish "how often prices change" figures until the history is rebuilt on one source per model.
Code
src/pages/models/[id].astro

Benchmark score on a 0 to 100 scale

Used on: Leaderboard, Benchmarks.

Puts benchmarks with different scales on one scale so they can be combined.

(score − scale minimum) ÷ (scale maximum − scale minimum) × 100, flipped for benchmarks where lower is better, then limited to the range 0 to 100.

Limits
Depends on the scale limits we set for each benchmark.
Code
src/data/hub-dashboard.ts

Evaluated composite score

Used on: Leaderboard.

One score per model from the benchmarks we hold for it, marked down when the evidence is thin.

Benchmark score = sum of (normalised score × benchmark weight) ÷ sum of the weights covered. Composite = benchmark score × (0.7 + 0.3 × coverage factor) × evidence factor. Coverage factor = the share of total benchmark weight covered ÷ 0.55, capped at 1. Evidence factor = 1.00 for seven or more scores, 0.98 for six, 0.96 for five, 0.93 for four, 0.88 for three, 0.82 for two. A model needs at least two benchmarks to be scored.

Limits
Only as good as the scores underneath, most of which are labs' own claims and many of which lack a source link today. Arena ratings no longer count towards this score.
Code
src/data/hub-dashboard.ts

Value score

Used on: Best value.

Benchmark performance per dollar.

Benchmark score ÷ blended price × 10, rounded to a whole number. Zero when there is no price.

Limits
Inherits every limit of the benchmark score and the blended price.
Code
src/data/hub-dashboard.ts

Freshness score

Used on: Leaderboard.

A simple age band for a model.

Days since release: up to 30 = 100, up to 75 = 95, up to 120 = 90, up to 180 = 82, up to 270 = 68, up to 365 = 55, up to 540 = 40, older = 24, no date = 35.

Limits
Age is not quality. The release date carries the caveat described under the OpenRouter source.
Code
src/data/hub-dashboard.ts

Reliability floor

Used on: Reliability floor.

A score that cannot be carried by one strong area if a critical area is weak.

Each dimension score is the mean of its normalised source scores. Floor = the lower of (the weighted average of all scored dimensions) and (the weakest critical dimension + 10). A model is only ranked when every critical dimension has a score.

Limits
Few models have evidence in every critical dimension, so most are unranked.
Code
src/data/reliability-floor.ts

Staleness rules

Used on: Across the site.

When we stop treating a figure as current.

A price is flagged when it is more than 48 hours old. An active model is moved to "tracking" when its price is more than 30 days old and it was released more than 30 days ago. A speed figure is withdrawn when it is more than 30 days old.

Limits
These rules read the time a figure was last written. For hand-entered prices that time is refreshed on every run, so the rules do not catch them. This is a known fault. Speed figures are no longer shown, so the speed rule has nothing to act on.
Code
scripts/check-staleness.ts, scripts/normalize-model-registry.ts

The rules we write to

  • Each claim in an article links to its source and carries a date.
  • A number we calculated links to its formula on this page.
  • Where the data is partial, the article says so next to the number, not in a footnote.
  • A lab's own benchmark score is described as the lab's claim.
  • We do not publish anything built on a source whose terms we have not cleared.
  • We do not invent quotes or figures. If we cannot source it, it does not go in.
  • Articles are drafted with AI help, then checked against the data by a second, separate AI system before they are published. The figures come from the script above, not from a language model's memory.

Found something wrong?

Open an issue on GitHub with the page and the figure. We correct the data, note the change here, and fix any article that quoted it. See also the collection policy.