Draw the system around the model
A chat interface is only one entry point. An agent may also consume search results, uploaded files, messages, retrieved documents, and tool responses. It may then act using an identity with access to systems the original content author could never reach.
Map those paths before writing test cases. For each input, record who can influence it. For each tool, record the action it permits, the identity it uses, and the control that decides whether the action is allowed.
Treat retrieved content as untrusted data
Prompt injection can arrive through an ordinary user message or indirectly through external content. A document retrieved for summarization can contain instructions that conflict with the user’s task. OWASP describes these routes and recommends controls beyond text filtering, including tool validation and least privilege. Read OWASP’s prompt injection guidance ↗
Test whether the application preserves the difference between a user’s authorization and a statement found inside a document. A document that claims an action is approved should not be able to grant approval.
An assistant reads a synthetic support ticket containing a request to export a separate test customer’s records. The expected result is that authorization controls prevent the export, regardless of what the model proposes.
Evaluate the tool, identity, and approval together
OWASP’s excessive agency guidance highlights three sources of risk: unnecessary functionality, broad permissions, and too much autonomy. A read-only task should not inherit deletion capabilities simply because an integration exposes them. Explore OWASP’s excessive agency guidance ↗
Review permissions at the downstream system. Check whether actions run with the requesting user’s access or a shared privileged account. For consequential changes, verify that any required approval is bound to the actual action, target, and parameters before execution.
A persuasive model response is not evidence that an operation occurred. Conversely, a polite refusal does not prove that no tool call happened. Inspect the execution trace and the resulting state.
Build a small, useful test matrix
- Source: direct user input, a retrieved document, or a tool response.
- Boundary: one user, tenant, session, or tool permission crossing into another.
- Action: read, update, export, send, or delete, using synthetic targets.
- Expected control: server-side authorization, constrained arguments, or explicit approval.
- Observed result: proposed action, executed action, returned data, and resulting state.
- Repeatability: test configuration, model version, attempt count, and observed variation.
Include normal tasks alongside adversarial cases. A control that blocks every action may prevent misuse while also breaking the product. Measure both kinds of behavior within a defined test budget.
Report the boundary that failed
A useful finding names the attacker-controlled input, the affected identity, the tool or data boundary crossed, and the business consequence demonstrated with test data. Record enough configuration to repeat the case, while keeping credentials and sensitive content out of the report.
Describe outcomes within the tested conditions. A blocked example is evidence about that example, not proof that prompt injection has been eliminated. OWASP recommends recording the limits of evaluations and retaining independent tool permission checks. Review evaluation guidance ↗
The practical goal is to keep an unexpected model decision from becoming an unauthorized system action—and to demonstrate where that control holds.
Make your assessment answer the right questions.
Bring your environment, your goals, and the decisions you need to make. Let’s define the testing that will help.
Talk about your project