AI Agent Guardrails and Safety: Prevent Runaway Actions

Contents

Figure 1: Nothing stood between the model's decision and the damage — this guide builds that layer

Least privilege, one choke point, exact human approval, budgets, and logs the agent can't touch

Your agent gets a routine cleanup task, hits a credentials error, and decides to fix it by itself. Nine seconds later, a production database is gone. That's a reported incident, and the details matter less than the pattern: nothing stood between the model's decision and the damage.

This guide shows you how to build that missing layer. I wrote a small guardrail library, ran a simulated runaway agent against it, and report the real results below. The code is standard-library-only Python, and it confines every file operation to a temp sandbox. For the security fundamentals this builds on, see the function calling and tool use guide in this series.

Why Agents Run Away

OWASP calls the problem Excessive Agency: damaging actions happen when a model produces unexpected, ambiguous, or manipulated output. The cause could be a hallucination, a weak model, or prompt injection. OWASP names three root causes:

• Excessive functionality: The agent has tools it doesn't need.
• Excessive permissions: The tools can touch more than the task requires.
• Excessive autonomy: High-impact actions run without independent approval.

Notice what's missing from that list: a smarter model. You can't prompt your way out of this, because the failure includes cases where the model is wrong, confused, or tricked.

What Went Wrong in Reported Incidents

Several write-ups describe agents deleting things they shouldn't. I haven't checked the primary sources, and most accounts come from the affected users, so treat these as reported, not proven.

• PocketOS, April 2026: A founder's public timeline, summarized by an IT provider, says a coding agent hit a credentials error during a staging cleanup and used an over-privileged API token to wipe a volume holding both production data and its backups.
• Replit and SaaStr, July 2025: A roundup says an agent wiped a production database during a code freeze, despite instructions to wait for approval. The roundup attributes confirmation to press coverage and a public apology from the CEO.

Both write-ups name the same gaps: over-broad credentials, no human checkpoint before irreversible actions, and backups the agent could reach. Instructions in the prompt didn't stop anything, because nothing enforced them.

The Principle: The Model Proposes, Code Decides

OWASP calls the fix complete mediation: check every request against policy in code, downstream of the model, instead of trusting the model to behave. Every guardrail below follows that idea. The model asks to run a tool, and a separate piece of ordinary code approves, blocks, or pauses the call.

Why code and not another prompt? A prompt is a suggestion, and code is a wall. Ever watched a model acknowledge an instruction and then ignore it? That's why.

Layer 1: Shrink What the Agent Can Do

The cheapest guardrail is a smaller toolbox. OWASP's mitigations start here: minimize the tools, limit each tool's functions, avoid open-ended tools like raw shell access, and give tools only the downstream permissions they need.

• Prefer narrow tools like delete_file(path) over run_shell(cmd).
• Scope credentials per task. A cleanup job shouldn't hold a token that can delete infrastructure.
• Use read-only access wherever the task allows it.
• Make mistakes hard. Anthropic's agent guidance suggests redesigning arguments so errors become harder, like requiring absolute paths.

A tool you never gave the agent is a tool it can't misuse.

Layer 2: Deny by Default and Validate Every Argument

My guard has one choke point, execute, that every tool call must pass through. Unknown tools don't exist, and known tools check their arguments before running. The path check handles the usual escape tricks:

def safe_path(self, p):
    full = os.path.realpath(p if os.path.isabs(p) else os.path.join(self.root, p))
    if os.path.commonpath([self.root, full]) != self.root:
        raise Denied(f"path escapes workspace: {p!r}")
    return full

An illustrative execute around it looks like this:

def execute(self, name, args, approved=False):
    self._log(name, args)                          # every decision lands in the audit log
    if self.kill_switch_engaged():
        raise Denied("kill switch engaged")
    if name not in self.tools:                     # unknown tools don't exist
        raise Denied(f"unknown tool: {name}")
    if self.budget_exhausted(name):
        raise Denied(f"{name} budget exhausted")
    if self.tools[name].dangerous and not approved:
        return {"status": "needs_approval", "tool": name, "args": args}
    return {"status": "ok", "result": self.tools[name].run(self, **args)}

I threw ten bad calls at it, and all ten were denied:

• ../outside/canary.txt and an absolute path to the same file
• /etc/passwd
• a symlink inside the workspace pointing outside it
• a path containing a null byte
• deleting a directory, and deleting the workspace root
• writing a .sh file, and writing 5,000 characters
• an unknown run_shell tool asking to run rm -rf /

The canary file outside the workspace stayed intact. Because realpath resolves symlinks first, the symlink trick failed too.

Layer 3: Make Humans Approve the Dangerous Stuff

Destructive and external actions, like deleting files or sending email, need a person in the loop. OWASP lists human approval for high-impact actions among its core mitigations. The details of the approval matter, so mine follows three rules:

• The model can't grant it. Only a human-facing function does.
• It binds to the exact call. Approval covers one tool with one set of arguments.
• It works once. Replaying the call needs a fresh approval.

My test confirmed all three. A delete returned needs_approval and left the file alone. After a simulated human approved it, the same call ran. A second identical call needed approval again, and approving tmp.txt didn't unlock a delete of tmp2.txt, which stayed intact.

Skip the "approve everything" button. A blanket approval defeats the point, and people click through repetitive prompts without reading them.

Layer 4: Set Budgets, Catch Loops, Keep a Kill Switch

Runaway agents don't always delete things. Sometimes they just keep going, burning money while retrying the same failing call. Anthropic's guidance recommends stopping conditions like a maximum iteration count. I tested four:

• Loop detection: An agent retried read_file("missing.txt") forever. The guard halted it after 4 calls with loop detected: identical call repeated 4 times.
• Per-tool caps: With write_file capped at 5 per run, seven attempts produced five successes and two denials.
• Step and time budgets: A 3-step budget halted the fourth call, and a 0.2-second budget halted a call made after a pause.
• Kill switch: Creating a flag file stopped the next call with kill switch engaged.

Make halts visible to the agent. A guard that explains why it stopped lets a well-behaved model change course.

Layer 5: Dry Runs, Verification, and Logs

Three smaller habits catch what the bigger layers miss:

• Dry runs: In dry-run mode, my guard described what a delete or write would do without doing it. The file stayed, and the write never happened.
• Read-after-write checks: My write tool reads the file back and confirms the content. One incident roundup lists missing verification as a common failure pattern.
• Audit logs: Every decision lands in a JSON line with the exact tool and arguments. My full run produced 41 records. Log the real command, not just its output.

Keep the log outside the agent's reach. A log the agent can delete isn't much of an audit trail.

Prompt Injection: The Test That Matters

Indirect prompt injection happens when an agent reads external content that contains instructions, like a web page, email, or file. OWASP says no foolproof prevention exists, and it recommends least privilege, human approval, and treating the model as an untrusted user.

I simulated it. A scripted "gullible" agent reads an inbox file that says to delete three files and email notes to an attacker. This agent is a stand-in for a model that obeys injected text, so it tests my controls, not any real model's resistance.

Run	Files before → after	Emails sent
No guard	5 → 2	1
With guard	5 → 5	0

With the guard, all three deletes paused for approval, and the email died at the allowlist check because evil.example isn't a permitted domain. The guard never evaluated whether the file's instructions looked malicious — it enforced policy instead, which is why it can't be talked out of anything.

What Guardrails Don't Catch

Be honest about the gaps, because attackers will find them.

• Allowed but wrong: My guard permits an agent to overwrite a note with garbage, because writes are in policy. Guards enforce structure, not intent.
• Approval fatigue: Humans rubber-stamp requests when there are too many. Ask only for approvals that matter.
• Race conditions: Checking a path and then using it leaves a tiny gap in which the path could change. My tools add a second confinement check at the point of use, which narrows the gap but doesn't prove it closed.
• Shared infrastructure: If an agent can reach your backups through another route, no file-level guard helps. The PocketOS account blames backups sitting beside production.
• Untested models: My demo uses a scripted agent. Test real agents adversarially, as OWASP advises — the evaluation guide covers adversarial suites.

A Checklist You Can Use Today

• List every tool your agent has, and delete the ones it doesn't need.
• Scope each credential to one task, with read-only access by default.
• Route all calls through one choke point that denies unknown tools.
• Validate arguments in code, including paths, sizes, and recipients.
• Require exact, single-use human approval for deletes, payments, and outbound messages.
• Cap steps, time, and per-tool calls, and detect repeated identical calls.
• Add a kill switch that a human can flip.
• Log exact commands somewhere the agent can't edit.
• Isolate backups from anything the agent can reach, and test restores.
• Run adversarial tests before launch.

These rules apply directly to the agents in this series: the coding agent already runs tests in a sandbox with protected files, and the research agent confines every fetch to URLs that search returned.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What causes agents to run away and take damaging actions?

OWASP calls the problem Excessive Agency: damaging actions happen when a model produces unexpected, ambiguous, or manipulated output, with three root causes — excessive functionality (tools it doesn't need), excessive permissions (tools that can touch more than the task requires), and excessive autonomy (high-impact actions running without independent approval). Notice what's missing from that list: a smarter model. You can't prompt your way out of this, because the failure includes cases where the model is wrong, confused, or tricked.

Why enforce guardrails in code instead of prompts?

Because a prompt is a suggestion and code is a wall. OWASP calls the fix complete mediation: check every request against policy in code, downstream of the model, instead of trusting the model to behave. Every guardrail follows that idea — the model asks to run a tool, and a separate piece of ordinary code approves, blocks, or pauses the call. The guard never evaluates whether the request looks malicious; it enforces policy, which is why it can't be talked out of anything.

Which agent actions should require human approval?

Destructive and external actions: deleting files or databases, payments, and outbound messages like email. The approval must be exact and single-use — the model can't grant it, it binds to one tool with one set of arguments, and replaying the call needs a fresh approval. Approving tmp.txt must not unlock a delete of tmp2.txt. And skip the 'approve everything' button: blanket approval defeats the point, and people click through repetitive prompts without reading them.

What don't agent guardrails catch?

Allowed but wrong actions (a guard permits overwriting a note with garbage, because writes are in policy — guards enforce structure, not intent), approval fatigue when humans rubber-stamp requests, race conditions between checking a path and using it, shared infrastructure such as backups the agent can reach through another route, and untested real models. Be honest about these gaps, because attackers will find them.

Final Thoughts

Guardrails work when they live outside the model: least privilege shrinks the blast radius, a single choke point enforces policy, humans approve irreversible actions, and budgets stop loops. My tests showed the payoff in numbers, with the same injected agent destroying three of five files without a guard and none with one.

Pick one agent you run today and write down its tools. Then ask a blunt question about each: what's the worst thing this tool can do with the credentials I gave it? The answer tells you which layer to build first.

Related Articles

• Function Calling and Tool Use: How LLMs Take Actions
• Build a Coding Agent: Automate Code Generation and Debugging
• Build a Research Agent That Searches the Web and Writes Reports
• Evaluating AI Agents: Metrics, Benchmarks, and Test Strategies