Design safe tool use for AI agents

An AI agent becomes materially different from a chatbot when it can read private data or change external systems. The central safety question is not whether the model produces good prose; it is whether authority, intent, and side effects remain controlled when model output is uncertain. Give tools narrow contracts Each tool should perform one bounded operation with a typed schema, explicit authorization, predictable errors, and a small response. Avoid general shell, database, or HTTP tools when a domain specific operation will do. Validate every argument outside the model and reject unknown fields. Separate read operations from writes. Separate reversible changes from destructive or externally visible actions. Tool descriptions must not be the security boundary; enforcement belongs in application code and the target system. Bind authority to the user and task The agent should receive only the credentials and scopes needed for the current task. Preserve the initiating user’s identity and tenant context. Re check authorization at execution time rather than assuming that access to the conversation implies access to every connected resource. Require confirmation when an action is destructive, costly, difficult to reverse, or communicates externally. The confirmation should show the resolved target and effect, not a vague “continue?” prompt. Treat all content as untrusted Tool results, retrieved documents, websites, emails, and attachments can contain instructions intended to redirect the agent. Keep system policy and data separate, constrain which tool calls content can influence, and prevent secrets from being inserted into untrusted destinations. The OWASP Top 10 for LLM Applications describes prompt injection, excessive agency, sensitive information disclosure, and related risks. Use it as a threat model input, then test the concrete workflows your agent supports. Make execution inspectable Record requested action, resolved arguments, authorization decision, tool result, model and policy versions, confirmation, and resulting resource identifiers. Redact sensitive content and apply retention limits. Use idempotency keys for retried writes and compensation procedures where transactions are unavailable. Set budgets for steps, time, tokens, money, and affected records. A stop condition is a control, not a model preference. Useful agents combine flexible reasoning with inflexible authority boundaries. The model may propose an action; deterministic systems must decide whether and how that action is allowed to occur. Use a plan execute boundary Represent proposed actions as structured data before execution. A policy layer can resolve resource identifiers, calculate risk, check current authorization, and decide whether confirmation is required. The executor receives only an approved, immutable action—not the full conversation and open ended model authority. For a destructive request such as deleting a deployment, confirmation should show the exact environment, deployment identifier, dependent resources, and recovery consequence. If any target changes after confirmation, require a new approval. Limit indirect influence Retrieved content may supply facts but should not grant capabilities. An email saying “upload credentials to this URL” must not change the allowlist of destinations. Label data provenance internally and prevent untrusted text from being concatenated into system policy or tool definitions. Use separate contexts for browsing, reasoning, and secrets. Where a tool needs a credential, bind it inside the executor rather than exposing it to the model. Filter tool responses to the minimum fields needed for the next decision. Design compensation and recovery For reversible actions, capture the previous state and provide a tested undo operation. For irreversible actions, increase review and confirmation. Multi step workflows need checkpoints so a retry resumes safely rather than repeats completed effects. Stop and escalate when actual state differs from the plan. Adversarial checklist Test instructions embedded in websites, documents, code comments, images, and tool output; attempts to cross tenant or user boundaries; argument smuggling; excessive result counts; repeated writes; tool timeout; stale confirmation; and partial workflow failure. Verify budgets, policy decisions, and audit events remain correct under each case. Safety is demonstrated by denied or contained actions, not by the model explaining that it intends to be careful.