AI Agent Prompt Injection Defense: Secure the Action Boundary

The most reliable prompt injection defense is not a perfect malicious-instruction detector. It is an architecture that assumes some manipulation will get through and still prevents the agent from performing an unauthorized action.

Treat every webpage, email, document, tool result, memory, and agent message as potentially adversarial data. Preserve where it came from. Give the agent only the capabilities required for the current task. Before any consequential write, disclosure, purchase, message, or execution, enforce authorization in deterministic code outside the model.

This reframes prompt injection from a wording contest into a security engineering problem.

Secure the action boundary, not just the prompt. Untrusted content may influence a model's plan, but it must never be sufficient authority to spend money, disclose data, change state, or expand its own permissions.

The useful threat model is source to sink

OpenAI describes (opens in a new tab) prompt injection using source-sink analysis. A source is any channel an attacker can influence. A sink is a capability that becomes dangerous in the wrong context. The exploitable path is the connection between them.

Common sources include:

• webpages, search results, and linked files;
• email bodies, attachments, and calendar invitations;
• repository content, issues, comments, and dependency output;
• retrieved documents and persistent memory;
• MCP tool results, connectors, plugins, and remote agents;
• user-supplied prompts copied from another person.

Common sinks include:

• sending a message or publishing content;
• uploading a file or disclosing a secret;
• modifying or deleting records;
• running code or shell commands;
• granting access or changing permissions;
• purchasing, refunding, transferring, or committing funds.

An agent that only summarizes a public document has an untrusted source but no powerful sink. A system that can send private files to arbitrary endpoints has both. Reducing source exposure helps, but closing, narrowing, or guarding the sink is usually the stronger control because it remains effective when classification fails.

This complements the broader principle that agents need boundaries. Prompt injection is one way an agent can be pushed off course. Mistakes, ambiguous instructions, compromised tools, and policy conflicts can reach the same dangerous action. A control at the action boundary contains all of them.

Why input filtering cannot carry the defense

Early injection examples looked like conspicuous commands hidden in a page: ignore previous instructions, reveal a secret, follow this link. Filters and model training can catch many such attacks. Real attacks increasingly resemble ordinary social engineering. They use context, plausible business language, false authority, indirect requests, and multi-step persuasion.

That makes perfect detection an unsafe production assumption. The system must distinguish a legitimate instruction quoted inside a document from an instruction addressed to the agent. It must detect misleading claims whose danger depends on private context. It must do so across languages, encodings, images, tool responses, and messages from systems that may themselves be compromised.

Model-level defenses remain useful. Train the model to separate data from instructions. Scan external content. Ask a second classifier to flag suspicious tool results. Use structured extraction to reduce raw text passed upward. These controls lower the probability of manipulation, but they are probabilistic. They should not be the only thing between hostile content and an irreversible side effect.

The secure assumption is simple: the model may eventually be convinced. The architecture must decide what happens next.

Put deterministic controls between planning and acting

Separate the agent into two conceptual planes:

1. The reasoning plane reads context, considers options, and proposes actions.
2. The execution plane validates and performs only authorized actions.

The model can participate in the first plane. It must not be the sole policy engine for the second. A tool proxy or policy enforcement point should receive a structured action request containing the target, operation, data classification, provenance, user intent, and authorization context. Code then permits, denies, narrows, or routes the request for approval.

Boundary	Deterministic control	Failure it contains
Data read	Scope and row-level access rules	Agent retrieves unrelated private data
File access	Resolved-path allowlist and read-only mounts	Symlink or path escape reaches host secrets
Network egress	Destination, method, account, and payload policy	Injected content exfiltrates data
State change	Typed command plus invariant checks	Agent writes an invalid or excessive change
Financial action	Amount, recipient, frequency, and approval limits	Manipulation triggers unauthorized spending
Communication	Recipient and disclosure check with preview	Private content is sent to a third party
Permission change	Independent authorization service	Agent expands its own capabilities

Treat tool calls as requests, not commands. Validate arguments against business invariants, not only JSON schemas. A refund tool should independently verify the order, identity, eligibility, amount ceiling, and duplicate status. A valid string in the amount field is not proof the refund is authorized.

Bind authorization to the user's original intent. If the user asked the agent to summarize an inbox, a retrieved email cannot silently expand that authority into sending attachments. If the task legitimately changes, the system should obtain a new explicit authorization rather than allowing content inside the source to redefine scope.

Preserve provenance through the whole system

Many systems label a document untrusted when it enters, then lose that label when its content is summarized, stored in memory, or relayed by a subagent. That is trust laundering.

Every fact and instruction derived from external content should carry provenance: source identity, retrieval time, trust class, transformation history, and relevant access classification. When a planner proposes an action, the execution plane can then ask whether sensitive data came from a private source, whether the destination is controlled by a different party, and whether the action is grounded in user-authorized evidence.

Structured handoffs help. A browsing subagent can return facts with URLs and confidence instead of raw page instructions. But do not assume a message becomes trusted because another agent produced it. The subagent may have been manipulated, and its output may contain a transformed payload. Trust should derive from enforceable identity, scope, and provenance, not from the word "agent."

Persistent memory deserves special treatment. A malicious sentence saved today can steer runs weeks later, after the original source has disappeared from the active context. Memory writes should require provenance, namespace isolation, access control, expiry or review policy, and a deletion path. This is one reason retrieval is not grounding: placing text in context does not make it true, safe, or authorized.

Contain the environment and minimize capability

Least privilege must be concrete. Give the agent a task-scoped token, a narrow filesystem view, restricted network egress, and the smallest useful tool surface. Keep credentials outside the model context and outside execution sandboxes where possible. Use read-only access by default. Separate read and write tools so a research task does not inherit mutation rights accidentally.

Anthropic's 2026 containment report (opens in a new tab) describes why repeated approval dialogs are not enough: its telemetry showed users approved roughly 93 percent of permission prompts. After introducing an operating-system sandbox for Claude Code, permission prompts fell by 84 percent because safe activity could happen inside a harder boundary. Those product-specific figures are not universal benchmarks. They illustrate two broader lessons: approval fatigue is real, and strong containment can reduce both risk and friction.

Network allowlists require more precision than domain names. An approved domain can expose file uploads, redirects, server-side fetches, or attacker-controlled accounts. Treat each reachable operation as a capability. Bind egress to method, path, authenticated account, payload class, and task. Block direct arbitrary URLs when a brokered fetch or upload service can enforce those properties.

Sandboxing is not one control. Combine process isolation, filesystem mounts, resource limits, secret separation, and egress policy. Prefer mature isolation primitives over custom barriers, then test the integration around them. A sandbox that prevents host access but permits arbitrary outbound uploads can still lose everything mounted inside it.

Design approvals for decisions humans can actually make

Human approval is appropriate when an action is consequential and intent cannot be proven automatically. It is weak when used for every low-level step or when the reviewer lacks the context to judge the request.

A useful approval shows:

• the exact action and target;
• the data that will leave or change;
• why the agent believes it is required;
• which source caused the request;
• a readable diff or preview;
• the policy and limits being applied;
• safer alternatives, including cancel.

Approve a coherent transaction, not an unexplained command. "Allow network?" is not enough. "Send these three named fields to this verified recipient to complete the request" gives the reviewer a decision they can evaluate.

For repeated workflows, use pre-authorized envelopes with hard limits: a named system, permitted operations, data fields, amount ceiling, expiry, and revocation. The user or administrator approves the envelope once. The execution plane then enforces it on every call. Anything outside the envelope stops or escalates.

High-impact actions should support staged commitment. Draft the email before sending. Prepare a database transaction before commit. Generate a deployment plan before production execution. This creates a final boundary where state, policy, and intent can be checked together.

Test paths, not just payload strings

A prompt injection test suite should vary both the attack and the available capability. A system that resists one phrase may fail when the same goal is spread across an email, attachment, and linked page.

AgentDojo (opens in a new tab) introduced a dynamic benchmark environment for evaluating tool-using agents against prompt injection tasks. Its enduring design lesson is to test utility and security together. A defense that blocks every tool call is secure but useless. A useful system must complete benign tasks while refusing attacker-induced side effects.

Build tests around source-to-sink paths:

• untrusted email to outbound message;
• retrieved page to credential access;
• repository file to shell execution;
• tool result to memory write;
• memory entry to permission change;
• subagent response to external upload;
• private document to a newly introduced destination.

For each path, assert the final state. Confirm that no message was sent, no secret was read, no record changed, or the required approval occurred. Inspect traces to identify which layer caught the attempt. Repeat with paraphrases, encoded text, images, multi-turn setup, benign distractors, and compromised trusted tools.

Use red teams, but do not turn the result into a single attack-success score. Report the model, harness, tools, permissions, environment, defense configuration, and trial count. Track utility regressions and false approvals alongside blocked attacks. The real target is constrained impact under adaptive pressure.

Prepare for failure and limit persistence

Assume a harmful action will eventually be attempted. Every consequential tool needs an audit record with actor identity, user authorization, source provenance, inputs, policy verdict, and resulting state. Alerts should focus on boundary events such as unusual egress, cross-tenant access, bulk reads, new destinations, permission changes, and repeated denied actions.

Build kill switches at several scopes: cancel one run, revoke one agent token, disable one tool, isolate one tenant, and pause the whole service. Make credentials short-lived and independently revocable. Keep backups and compensating actions for reversible state changes. Practice incident response with traces that security teams can query without exposing unrelated private data.

After an incident, remove poisoned persistent state, not only the triggering session. Search memories, cached retrieval results, shared workspaces, and downstream agent messages. Rotate any secret the agent could reach, even when logs do not prove exfiltration. Then convert the causal path into a regression test.

Google DeepMind's security framework (opens in a new tab) and the OWASP Agentic Security Initiative (opens in a new tab) both point toward defense in depth across the model, orchestration, tools, identity, and environment. No one layer receives unlimited trust.

Production prompt injection checklist

• Inventory every untrusted source and every consequential sink.
• Draw the allowed source-to-sink paths for each workflow.
• Keep the reasoning plane separate from deterministic execution policy.
• Bind actions to explicit user intent and task-scoped authorization.
• Preserve provenance across retrieval, summaries, memory, and agent handoffs.
• Use read-only access, narrow tokens, and minimal tools by default.
• Keep secrets out of model context and execution sandboxes where possible.
• Enforce filesystem boundaries and operation-aware network egress.
• Validate business invariants, not only tool argument schemas.
• Stage high-impact actions and show a specific approval preview.
• Test multi-step source-to-sink paths with final-state assertions.
• Log boundary decisions and provide scoped revocation and kill switches.
• Clean persistent state and add a regression test after every incident.

Sources

• OpenAI: Designing AI agents to resist prompt injection (opens in a new tab)
• Anthropic: How we contain Claude across products (opens in a new tab)
• Google DeepMind: Securing the future of AI agents (opens in a new tab)
• OWASP: Agentic Security Initiative (opens in a new tab)
• AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents (opens in a new tab)