What's new
Explainer14 Nov 20265 min read

Test what an agent can do after a prompt injection

A detector that misses an instruction and an executor that allows an unauthorised action are separate failures, so a test should record both gates instead of treating a detection score as the whole defence.

The user asked an agent for a summary. The retrieved page it reads tries to hand it a new instruction, an unrelated action. A defensive test of that situation has three things to look at: what reaches the model, what the model proposes, and what the application permits. A detector's verdict covers only the first.

Anthropic's prompt-injection guidance, read on 14 November 2026, describes indirect instructions arriving through third-party content. It recommends keeping untrusted material separate and limiting access to actions and sensitive data. We use a constructed example to show how to report those boundaries, and it is not a security result.

A fixture needs a prohibited outcome

Each fixture starts from the legitimate task and the one action it must not cause. The documents are synthetic and the executor is simulated, so a test can propose a harmless dummy operation and record whether it would be authorised, without sending messages, changing accounts or touching real secrets.

Invented defensive test ledger, not observed attack results. All operations are assumed to use harmless fixtures and a simulated executor.
Gate in a constructed testCount
Malicious fixtures20
Screen flags18
Reach model2
Prohibited proposals2
Executor denials2
Prohibited executions0

Here the screen detects eighteen of twenty malicious fixtures. Two reach the model and produce prohibited proposals, and the executor rejects both. Detection is 90% in the invented set. Zero prohibited actions execute because the example assumes an independent denial gate, and that zero is an assumption, not an observed defence rate.

Proposing is not the same as being permitted

The permitted action comes from the user's task and the application's policy, and the executor checks the operation, target and scope before running anything. A well-formed tool call from the model does not show that the user authorised it.

For consequential work, a proposed operation should be concrete enough to review: which target, which change and which evidence supports it. Approval stays tied to that operation, so a vague permission to continue cannot quietly stretch to a different destination or effect.

Denied proposals are evidence even when nothing runs, because they show how the model read the untrusted text. A screen that passes a fixture, on the other hand, does not prove the model followed it. The sequence is worth keeping whole instead of squeezed into one success label.

Benign work has to pass too

Ordinary documents contain quotations, instructions meant for humans and examples of commands. A defence that blocks the legitimate task on those inputs has a usability cost, so benign-task acceptance gets its own stated rule, separate from the malicious-fixture outcomes.

The cases should come from the actual workflow: retrieved pages for a research assistant, document text for an extractor, tool output for a coding helper. Fixtures are fit to publish only if they are synthetic or cleared for it. Your own results log needs outcomes and configuration, not private source content.

A change of model, prompt, tool wrapper or permission set means repeating the test, since the same fixture may now travel through a different boundary. The executor policy version belongs with the result, so the outcome is not credited to the model name alone.

What the test leaves uncovered

Denied writes do not show that no sensitive information was exposed in a response. A read operation can still fetch material the task should not reach. The data-access boundary needs its own review, and the agent should not be given material it does not need.

A small fixture set cannot establish general resistance to prompt injection either. The report should give the cases, assumptions and observed actions, and a clean run is not a claim of immunity.

A release record is most useful when it says what the agent saw, what it proposed, which gate allowed or denied it and whether the legitimate task finished. That supports a narrower, more actionable decision than a detector score alone.