Valid JSON is only the first pass for an AI extractor
An extractor can return a perfectly shaped object with the wrong amount inside it, so the response, the schema and the evidence need separate checks.
Take a document containing the amount 42.00. The schema requires a numeric amount and a currency. A response containing 24.00 passes the type check and still fails the job. No model benchmark is needed to see the gap: the schema describes the permitted shape, while the source document determines the correct value.
Structured outputs are good for defining that shape. Anthropic's documentation, read on 14 November 2026, describes constrained JSON output, supported-schema limits and exceptions for refusals and token-limit stops. Our proposed reporting method keeps those outcomes separate from factual acceptance.
HTTP success is not task success
The documentation says a refusal can arrive with HTTP status 200 and need not match the requested schema. A response cut short at max_tokens can also be incomplete, so the completion status needs inspecting before the body is treated as an answer.
Parsing and validation come next, then a check of the values against the document and the job's rules. Those stages give a better failure record than one success flag, and they show where to spend effort: a malformed result needs different attention from an accurately formatted wrong answer.
| Cumulative gate in a fictional run | Passing requests | Share of original run |
|---|---|---|
| Transport completed | 1000 | 100% |
| Finished with usable structured output | 980 | 98% |
| Passed schema checks | 970 | 97% |
| Passed task checks | 900 | 90% |
In the example, 970 requests pass the schema and 900 pass the task check. Task acceptance is 90% of the original run, or about 92.8% of the schema-valid subset. Both statements are true and answer different questions, so a percentage needs its denominator beside it.
Evidence comes before the prompt
Each extracted field needs something that supports it, decided in advance. A simple document might allow a page reference and the exact text span. A calculated total may need the input lines and the calculation. A reviewer can inspect evidence; the model's confidence proves nothing.
Missing information needs a representation too. If the amount is absent, an explicit missing value can be the correct result, and a schema that demands a number without allowing an absence state makes that outcome hard to express. Absent and ambiguous fields are worth testing before running a large collection.
Normalisation rules need defining as well. Open points include whether a date with no year stays incomplete, whether the extractor may convert a currency symbol into a code and which decimal notation is allowed. These are application decisions. During testing, the original text should sit beside the normalised value so the conversion can be checked.
Retries depend on what failed
A refusal should stay a refusal; a blind retry loop obscures it and adds spend. For an incomplete result, check the limit and request size before trying again. For a schema-valid wrong value, more output space does not address the evidence failure, and the prompt, document handling and acceptance rule need review.
Each original job gets one final result even if it has several attempts, and the spend on those attempts is counted together. Otherwise a high pass rate can hide the failed requests that made the answer possible.
The awkward documents deserve deliberate tests: two different totals, a revised version, a table continued on another page, a field that is mentioned but not filled in. Use examples appropriate to your work. Fixtures should only be public when they are synthetic or cleared for publication.
Where the thresholds sit
Usable completions, schema validity and task acceptance each need their own threshold. If a wrong field is especially costly, make that failure a release blocker even when the average looks acceptable. The reviewed examples and the exact schema version stay with the result.
Your own results log can then say where an extractor is reliable enough for the job and where a person still needs to check it. A green parser alone cannot make that decision.