What's new
Analysis14 Nov 20265 min read

The useful AI price is the cost per accepted answer

In our worked example, the candidate with the smaller API bill costs more per usable result, so count accepted answers before calling a model cheap.

Candidate A costs $8 for a test run. Candidate B costs $12. On spend alone, A wins. If you accept 800 of A's results and 960 of B's, A costs $10 per thousand accepted results and B costs $12.50. A still wins, but by less than its bill suggested. A third candidate spends the same $8 as A and passes only 500 cases, which puts its cost per thousand accepted results at $16.

Those are invented numbers. We have not run these candidates or assigned the figures to real models. What matters is the unit: spend divided by useful results. You can add it to your own results log without importing somebody else's benchmark or pretending your own work resembles theirs.

Illustrative counts and spend. Cost per 1,000 accepted results = total spend ÷ accepted count × 1,000. No retries or human costs are added separately.
Fictional candidateAttemptsAPI spendAcceptedCost per 1,000 accepted
Candidate A1000$8.00800$10.00
Candidate B1000$12.00960$12.50
Candidate C1000$8.00500$16.00

What counts as an accepted result

An accepted result is one that passes a rule you set in advance. For a document extractor, the rule might require the correct value and a passage that supports it. For a label, it might be an exact match to a reviewed answer. For a draft, it might mean the editor can publish it after a defined amount of checking. The rule needs writing down before the results arrive.

A parsing success is a different measure. Anthropic's structured-output documentation, read on 14 November 2026, describes schema-constrained responses and separate handling for refusals and token-limit stops. A format check does not prove that the extracted amount matches the document, so format validity and factual acceptance belong in separate columns.

For judgement-heavy work, the review note matters. “Good enough” becomes hard to compare when one reviewer means readable and another means ready to send. If the definition changes halfway through the test, mark the change and review the earlier examples again.

Failed attempts belong in the bill

Total API spend for the test counts, including attempts that produced nothing usable. Dividing only the spend on successful requests by the successful count quietly removes the price of failure. Retries add to the spend too.

The denominator needs care. A workflow that tries one document three times has made three attempts at one job, and if the final result is accepted that is one accepted job. A comparison with no retries is simpler, with one attempt per case. Mixing the two definitions and calling the result an improvement would mislead.

A test with zero accepted results has no finite cost per accepted result, and the report should say so. Turning it into $0 because a division failed would make the worst candidate look free.

Limits an average hides

A lower cost per accepted result can still be unsuitable. You may need a response within a particular time, or have one kind of error you will not accept. Those conditions sit beside the cost; an average should not trade them away.

Human review can change the picture again. If you measure it, report the time before putting a money value on it, along with the hourly rate you assume and whether you timed the checking or estimated it. An API bill is observable; a guessed labour saving is an assumption.

The case count belongs in the report too. Passing a tiny hand-picked set does not establish a stable rate for the next month's work. A held-back set, one you did not use to improve the prompt, gives a fairer picture, and the failures are worth retaining. If the application meets a new kind of document, add a category rather than silently replacing difficult old examples.

Which fields let you recompute the cost

The row to keep holds total spend, job count, attempt count, accepted count and the acceptance rule, with model and route identifiers, prompt version and test date. The cost can be derived from those, leaving the original counts visible, so a later comparison still works if your preferred display unit changes.

When a model next cuts its token price, rerun the jobs. What matters then is how much it costs to produce an answer you can use.