Choose an AI model by the job you need finished
A model can be cheap to run and still be wrong for your work, so start with a job you can mark and use the price list only to narrow the field.
Labelling messages, extracting fields from a document and drafting a report are three different jobs. Treating them as one buying decision hides the differences. The label might be a few tokens. The report might be longer than its instructions. The extraction has to preserve a value exactly, even when the document is untidy. Each job gets its own shortlist.
We suggest a small written brief for each: what goes in, what must come out, what counts as a failure and how long the person waiting can reasonably wait. It belongs next to your own results log. Without it, a new release becomes a reason to switch before anyone has said what it needs to improve.
Price depends on the shape of the work
The table uses invented workloads and assumed rates of $1 per million input tokens and $5 per million output tokens. These are teaching rates, not a quote for a named model. The same pair of rates gives quite different bills depending on what the job reads and writes.
| Illustrative job | Input tokens | Output tokens | Cost per request |
|---|---|---|---|
| Label messages | 2000 | 50 | $0.00225 |
| Extract document fields | 8000 | 400 | $0.01000 |
| Draft a report | 2000 | 4000 | $0.02200 |
| Read a long dossier | 100000 | 1000 | $0.10500 |
The dossier spends most of its budget on input and the report spends most on output, so a ranking by output price alone would miss the difference. Before choosing a cheaper model, count representative requests from your own application, including the instructions and documents it sends. A range is worth noting as well as the typical case, so one unusually large file does not come as a surprise.
A shortlist is mostly exclusions
For extraction, first check that the API supports the format you need, then test missing fields and ambiguous values. For drafting, decide what the editor will accept before comparing the prose. For coding, write down the command or test that proves the change works. These are suggested pass conditions, not claims that one model family is better at a task.
Long documents need a capacity check before any comparison. Anthropic's context documentation, read on 14 November 2026, counts the request and new output within the window, including tool definitions. A document that fits on its own may leave too little room for the rest of your request, so reserve that space explicitly.
Support for a feature only gets a model onto the shortlist; it says nothing about quality. A model can accept a long file and still not give the answer you wanted. Two columns help: one for advertised support and one for results you have checked yourself, which stays at “not tested” until you have evidence.
Failures tell you more than scores
Every candidate should see the same inputs, with the prompt and settings stored alongside the results. Include cases you expect to be awkward: an empty document, contradictory instructions in quoted material, a missing answer, a request that needs a refusal. Pick them because they matter to your application, not because they make a favourite model look good.
A failed example is worth keeping, with a note on what went wrong. A wrong value, a broken format and a slow but correct answer call for different fixes, and a single total score hides the difference. If one failure would stop the job going live, treat it as a separate release condition.
For a close comparison, hide the model names while reviewing the output. Where the decision needs judgement, a short note giving the reason for acceptance is more use to the next reviewer than a preference recorded as a percentage.
A decision you can reverse
The log entry should name the exact model, provider route, prompt version, test set and date checked, plus token usage, measured waiting time and the number of results you accepted. A price table helps estimate spend, but it cannot supply those observations.
Pick a candidate for this job, under these conditions, and hold on to the runner-up and the failed cases. When a new model appears, rerun the comparison against that record. The comparison then comes down to whether the replacement finishes more of our work, within the limits we set.