Skip to content
Amin Mirlohi
4 min read

Multi-Agent Systems Need Boundaries, Not Bigger Prompts

When an agent system misbehaves, the instinct is to add more instructions. Usually the real fix is structural: explicit states, hard guards, and small tools with narrow contracts. Reliability is an architecture decision, not a prompting one.

#Agents#Reliability#Architecture

Most "the agent went off the rails" problems are not prompting problems. They are architecture problems wearing a prompting costume. The reflex, when an agent loops, hallucinates a tool call, or quietly does the wrong thing, is to add another paragraph of instructions. Sometimes that helps. More often you are negotiating with a system that has no enforced boundaries, and negotiation does not scale.

The reliable move is to take decisions away from the model and give them to the system. Bigger prompts ask the model to behave. Boundaries make misbehavior impossible, or at least observable.

Why "just prompt it better" stops working

A prompt is a soft constraint. It biases the model's distribution; it does not enforce anything. That is fine for tone and formatting. It is a bad place to put your correctness guarantees, because a soft constraint fails silently and probabilistically: exactly the failure mode you cannot debug from a log after the fact.

As you add tools, memory, and multiple agents, the space of things that can go wrong grows combinatorially while the prompt stays a flat wall of text. You end up with a 2,000-word system prompt that is really an unversioned, untested specification. I have written that prompt. It does not age well.

Boundaries that actually hold

Three structural patterns do most of the work.

1. Make the control flow explicit

An agent "deciding what to do next" in free text is a state machine with the states left implicit. Make them explicit. When the legal transitions are enumerated, an illegal one is a caught error instead of a weird trajectory you discover in production.

from enum import Enum
 
class State(str, Enum):
    PLAN = "plan"
    RETRIEVE = "retrieve"
    ACT = "act"
    VERIFY = "verify"
    DONE = "done"
 
# Only these transitions are allowed; anything else is a bug, not a "decision".
TRANSITIONS = {
    State.PLAN:     {State.RETRIEVE, State.ACT},
    State.RETRIEVE: {State.ACT, State.VERIFY},
    State.ACT:      {State.VERIFY},
    State.VERIFY:   {State.ACT, State.DONE},
}
 
def step(state: State, proposed: State) -> State:
    if proposed not in TRANSITIONS[state]:
        raise GuardViolation(f"illegal transition {state} -> {proposed}")
    return proposed

The model can still propose the next state. It just cannot invent a transition that your system never sanctioned.

2. Give tools narrow contracts

A tool that can do anything is a tool you cannot reason about. Prefer small tools with typed inputs, validated outputs, and side effects that are idempotent or explicitly gated. A search(query: str) -> list[Doc] you can test; an execute(arbitrary_code) you can only hope about.

The expensive lesson

The blast radius of an agent equals the blast radius of its most powerful tool. Audit your tools by what they can do on their worst day, not their best.

3. Put a deterministic verifier after the model

The cheapest reliability upgrade in most agent systems is a non-model check between "the agent produced output" and "the system acts on it." Schema validation, a policy check, a sanity assertion on the result. Deterministic guards catch the long tail of weird outputs that no prompt will reliably prevent, and they fail loudly.

The founder angle

There is a product reason to care, not just an engineering one. The thing customers actually pay for in an AI product is dependability: the same input behaving the same way on Tuesday as it did on Monday. Demos reward capability; production rewards consistency. Boundaries are how you convert an impressive demo into something you can put your name on.

It also changes how a team builds. When control flow is explicit and tools have contracts, you can write tests, you can hand a component to someone else, and you can change one piece without the whole thing trembling. A monolithic prompt resists all three.

What this looks like in practice

Start from the smallest system that can possibly work (one agent, two tools, an explicit loop with a hard iteration cap) and add capability only when an evaluation tells you it is needed. Every new power gets a boundary at the same time: a state, a contract, a guard. The goal is not to constrain the model for its own sake. It is to make the system's behavior legible, so that when something breaks you can point at the part that broke.

Capable models are abundant now. Reliable systems built on top of them are not. The difference is almost never a better prompt. It is boundaries.

Related reading

← All writing