Can you trust AI agents? Guardrails and hallucinations

Here's the honest answer first: no, you can't trust an AI agent the way you trust a calculator. It will be confidently wrong at some point. Every model does it, and no prompt fully fixes it.

But that's the wrong question. The useful question is: when it's wrong, what is it allowed to do? AI agent guardrails are the answer to that. They don't make the model smarter. They make its mistakes small, visible and reversible.

The fear I hear from founders is specific: it'll tell a customer something false, or do something that can't be undone, and we'll be the ones held responsible. That fear is reasonable. The fix is mostly design, not magic.

You're responsible for what your bot says

Start with the part that isn't technical. In Moffatt v. Air Canada, decided in February 2024, the airline's website chatbot told a customer he could claim a bereavement fare after booking. That wasn't the airline's policy. The tribunal held Air Canada liable and ordered it to pay C$812.02.

The amount was small. The reasoning wasn't. Air Canada argued the chatbot was responsible for its own actions, and the tribunal called that "a remarkable submission", adding that "it should be obvious to Air Canada that it is responsible for all the information on its website."

That's a Canadian small claims decision, not a global rule, and I'm not a lawyer. But the principle is the one to plan around: if your agent says it, you said it.

Why AI agents hallucinate, and why prompts won't fix it

A language model predicts plausible text. Most of the time plausible and true line up. Sometimes they don't, and the model has no internal alarm that goes off when it's making something up. It sounds exactly as sure either way.

You can reduce this with better instructions and better context. You can't remove it. So the working assumption has to be: treat everything the model produces as untrusted input, the same way you'd treat text a stranger typed into a form on your website.

That shift changes the design. You stop asking "how do I make it never wrong?" and start asking "what checks does its output pass through before anything happens?"

Keep the action space small

The most effective guardrail is also the most boring one: give the agent less to do.

In an Ask HN thread on preventing hallucinations in production, one practitioner put it plainly: "The biggest reduction in hallucinations usually comes from shrinking the action space, not improving the prompt." That matches what I see.

In practice:

• Fewer tools. Every tool you add is another way to be wrong. Give the agent the three it needs, not the thirty it might use.
• Narrow permissions. If it only needs to read orders, it gets a key that can only read orders.
• Read-only by default. Writing, sending, deleting and paying are each a separate, deliberate decision.

The security world has a name for getting this wrong. The OWASP Top 10 for LLM applications lists Excessive Agency as LLM06: an agent with too much functionality, too many permissions or too much autonomy. Prompt injection, where hidden instructions in content trick the model, is number one on the same list. A small action space limits the damage from both.

The model proposes, your code decides

The same HN thread has the line I'd put on the wall: "LLMs generate proposals, deterministic systems enforce invariants."

In plain terms: let the model suggest, and let ordinary code check the suggestion against rules that never bend. For example:

• The model drafts a refund. Code checks it's under the limit, the order exists, and it hasn't already been refunded.
• The model extracts figures from a document. Code checks the totals add up and the fields are in the right format.
• The model writes a reply to a customer. Code checks it doesn't mention prices, dates or policies that aren't in your approved source.

The rules live in the system, not in a sentence in the prompt asking the model nicely. A prompt is a request. Code is a guarantee.

When should a human approve what an AI agent does?

Human-in-the-loop sounds like a slogan, so here's what it means concretely. OpenAI's practical guide to building agents names two triggers for bringing a person in:

1. Failure thresholds. Set limits on retries. If the agent keeps failing, for example it can't work out what a customer wants after several attempts, it hands over to a human.
2. High-risk actions. Anything sensitive, irreversible or high-stakes waits for a human, at least until you trust the agent's track record. The guide's examples are cancelling orders, authorising large refunds and making payments.

Anthropic's guidance on building effective agents says something similar: agents "can then pause for human feedback at checkpoints or when encountering blockers", and it's common to add stopping conditions such as a maximum number of steps.

A simple way to sort your own actions:

• Reversible and low value: the agent does it and logs it.
• Reversible but customer-facing: the agent drafts, a person approves, at least at first.
• Irreversible or high value: a person approves, always.

The AI agent guardrails most people skip

Guardrails aren't one thing. OpenAI's guide describes them as layers: relevance classifiers that catch off-topic requests, safety classifiers that catch jailbreaks and prompt injection, simple rules like blocklists and input length limits, and risk ratings on each tool. No single layer is enough. Together they catch most of what matters.

Around those, there's the operational work:

• Sandbox testing. Anthropic recommends "extensive testing in sandboxed environments, along with the appropriate guardrails." Test with fake data and fake accounts, and include the messy cases, not just the happy path.
• Logs and an audit trail. When something goes wrong, you should be able to see every step the agent took and why, in minutes.
• Cost monitoring. A loop that retries forever is a reliability problem and a billing problem. Set limits and alerts.

None of this is glamorous, and it's where most of the build time on a reliable agent actually goes. It's also what decides whether the agent is still working three months after launch, which I've written about in most AI projects don't fail during the build.

A short checklist before you let an agent near customers

• Can you list every tool it has and every permission each tool carries?
• Is anything it can do irreversible without a human saying yes?
• Are the business rules enforced in code rather than in the prompt?
• Does it escalate to a person after repeated failures?
• Has it been tested against bad, strange and hostile inputs?
• Could you reconstruct what it did yesterday from the logs?
• Would you be comfortable if a customer screenshotted its worst answer?

If you can't answer yes to all of these, it's not ready for customers yet. It might still be ready for internal use, where a mistake costs a colleague five minutes rather than a customer's trust.

Where to start

Pick the agent you're planning, or the one you already have, and write down every action it can take. Next to each, write what happens if it gets that action wrong. Anything with a bad answer in the second column either loses the permission or gets a human in front of it.

This is the part of building agentic systems I spend the most time on, because it's the part that makes the rest safe to use. If you want a second opinion on where your agent needs a human, send me what you're building.