An assistant that writes a suggestion can be wrong. An agent that acts on that suggestion can also change records, contact people or expose information. Adding tools changes the risk, even if the conversational interface looks exactly the same.
Designers and developers should treat agency as a set of deliberately granted capabilities, not a personality trait. The key question is not whether an agent seems trustworthy. It is what the system allows it to do when it misunderstands a request or encounters malicious content.
Treat retrieved material as data, not authority
Consider an illustrative research agent that reads a web page to summarize a product specification. The page contains a paragraph telling the agent to ignore its task and send internal notes elsewhere. That paragraph is part of the material being read. It is not an instruction from the person who authorized the research.
OWASP's Prompt Injection guidance distinguishes direct manipulation through prompts from indirect manipulation through external content. It also notes that retrieval and fine-tuning do not fully remove the risk. Instructions hidden in documents, websites or images can matter if the model processes them.
Do not rely on “ignore malicious instructions” as the only defense. The application needs boundaries that still hold when the model follows the wrong instruction.
Grant the smallest useful capability
A research task may need read access to an approved source collection. It does not necessarily need permission to email results, change sharing settings or inspect unrelated records. Keep those capabilities separate.
Use narrow tools with validated parameters instead of a single broadly privileged operation. The trusted application should check identity, scope and allowed destinations. A model's claim that an operation is authorized is not an authorization decision.
For designers, this means the interface should accurately show the agent's current scope. “Can read selected documents” is more meaningful than a vague “Connected” badge. For developers, the permission must be enforced behind that badge, including for requests made outside the visible interface.
Put consequential actions behind specific approval
If an agent proposes sending a summary, show the recipient, content and attachments before approval. If it proposes deleting records, show exactly what will be affected and whether recovery is available. Avoid confirmations that merely ask whether the person trusts the agent.
Bind approval to the reviewed action. If the recipient, content or target changes materially, the prior confirmation should not silently cover the new operation. Where permissions have expired or access has been revoked, recheck them at execution time.
Human approval is also not a complete defense. People can miss details, and an interface can conceal consequences. Use it with least privilege, validation and sensible limits rather than as an excuse to give the agent unrestricted access.
Plan for uncertainty after execution
A request can time out after an external system has already completed it. Automatically retrying may duplicate a message or transaction. Design an explicit unknown state and reconcile the result before repeating a consequential operation.
Record enough structured information to investigate actions without indiscriminately logging sensitive content. Separate what the agent proposed, what the person approved and what the tool actually reported. This distinction supports recovery and makes accountability clearer.
Test malicious retrieved instructions, unauthorized destinations, altered action parameters and partial failures. A successful happy-path demonstration says little about these boundaries. AI can generate attack ideas, but security review must examine the actual enforcement, not just the prompt wording.
Safe-agency checklist
- Separate external content from authorized instructions.
- Grant only the tools and data needed for the task.
- Validate identity, scope and parameters in trusted code.
- Show specific consequences before high-risk actions.
- Reconfirm materially changed actions.
- Reconcile unknown outcomes before retrying.
- Test adversarial inputs and denied operations.
Safe boundaries do not make an agent useless. They make its useful work easier to understand and contain. Designers make authority visible; developers enforce it; both remain responsible for the gaps between what was requested, proposed and actually done.
Sources
Key takeaways
- Treat retrieved content as untrusted dataInstructions found in documents, webpages or tool output must not acquire authority over the agent.
- Constrain capabilities outside the modelEnforce least privilege and authorization in trusted application and service layers.
- Require specific approval for consequential actionsMake the action, target and impact clear before execution, and plan for uncertain outcomes afterward.