There is a step in every AI transformation where the system stops
producing text a person re-keys somewhere else and starts doing the work itself. That step
is where most of the value is, and it is also where the risk profile changes completely.
An assistant that answers wrongly wastes someone's time. An agent that acts wrongly changes
a record. The engineering that separates those two outcomes is not about model capability —
it is about constraint.
What an agent actually is
Stripped of marketing, an enterprise agent is a loop: the model reasons about a goal,
selects a tool, observes the result, and repeats until the goal is met or a limit is hit.
In a system built for production that loop is defined declaratively — agents specified in
YAML rather than assembled in code — so that what an agent may do is reviewable by someone
who is not the person who wrote it. The runtime executes ReAct-style loops with tool access
that includes both built-in tools and, over MCP, the systems you already run.
That last point is the practical one for transformation programmes. The value of an agent
is proportional to what it can reach. An agent that can only read documents is a better
search box. An agent that can read a document, look up a record, draft a response and file
it is doing a job.
The three constraints that make autonomy approvable
Capability is the easy half. These three are what let a risk committee say yes.
Step and token budgets. Every run executes under an explicit ceiling, with circuit
breakers. This is what prevents an unbounded process from attaching itself to your systems.
It is worth being precise about the trade, because it is real: budgets prevent runaway
loops by truncating work. A task that genuinely needed more steps returns an incomplete
answer, and the budget is the reason. That is the correct trade for a production system,
and it is still a cost. A vendor who presents budgets as pure upside has not run one.
Scope isolation. Per-project isolation means an agent reaches only its assigned data. A
procurement agent cannot read HR records because the boundary is structural rather than
prompted.
The honest limit here matters too: per-project isolation makes some legitimate
cross-project questions impossible by design. Widening scope is an access-control
decision, not a configuration convenience — and treating it as the latter is how
organisations quietly dismantle the control they paid for.
Signed delegation. When one agent hands work to another, the hand-off is cryptographically
signed — Ed25519 in our implementation. This looks like over-engineering until you consider
the attack it prevents, which is the subject of the next section.
The security property nobody mentions in the pitch
Retrieval turns document ingestion into an attack surface.
Once a system retrieves documents into a model's context, anyone who can get a document
into the corpus can attempt to write instructions to the agent. A supplier's PDF, an
inbound email, a scanned form — any of these can carry text addressed not to the reader but
to the model.
This is why a serious request path runs an injection scanner before the model and an output
validator after it, and why agent-to-agent delegation is signed. Without authenticated
delegation, a successful injection escalates from a bad answer to a privilege problem: the
compromised agent asks another agent, with wider access, to do something.
Two honesty notes, both of which should appear in any vendor's material and rarely do.
No scanner is complete. Scanning reduces the attack surface; it does not close it. The
compensating controls are output validation, tool-level authorisation and the audit trail —
not the scanner alone. And this is a property of retrieval architectures generally, not
a weakness unique to any one product. The publishable claim is that it is treated as a
security boundary with named controls, not that it has been solved.
What agents cannot do
A technical evaluator reads vendor material looking for this section, and its absence tells
them something.
Retrieval returns the top k passages. Aggregation questions are not retrieval
questions. "How many of our contracts have an uncapped indemnity" requires every contract
to be examined, not ten passages to be found. A system that answers such a question from
retrieved passages will answer confidently and wrongly. Route these differently, or refuse
them explicitly.
Reranking cannot recover what retrieval missed. A cross-encoder improves the ordering
of the candidate set. If first-pass retrieval never surfaced the relevant document, no
amount of reranking will produce it.
Citations make error checkable, not impossible. A citation shows which passage an answer
was drawn from. It does not prove the answer follows from that passage. That is a
meaningful improvement — it converts an unverifiable claim into a verifiable one — but it
is not a correctness guarantee, and describing it as one is the most common overstatement
in this category.
Entity resolution is never fully solved. "Acme Ltd", "Acme Limited" and "ACME (UK)" may
or may not be one entity. A knowledge graph is only as good as that judgement, and the
judgement is domain-specific.
Parsing degrades predictably. Handwriting, low-quality scans, drawings without a vector
layer, and tables whose structure is carried only by visual alignment all reduce parsing
quality. Recovery is partial and should be measured per corpus rather than assumed. Anyone
quoting a global parsing accuracy percentage is quoting a number that was not measured on
your documents.
How to introduce agents without a governance fight
A sequence that works, in the order that keeps risk committees on side:
- Read-only agents first. Retrieval and drafting, with a human filing the output.
Establishes trust and produces the audit trail that later conversations will reference. - Reversible actions next. Creating a draft, adding a comment, raising a ticket —
actions where the worst case is noise, not damage. - Human checkpoints on consequential steps. Approval gates at the decisions that
warrant them, chosen by consequence rather than by uniform policy. - Irreversible actions last, and only where the audit record has been demonstrated to
reconstruct a run end to end: who triggered it, what context it used, what changed.
The framing that gets approval is not "how capable is this system". It is "what is the
maximum damage a single run can do, and how would we know?" An architecture that can
answer that question concretely will be approved. One that answers it with assurances will
not — and should not.
Next: AI transformation in regulated sectors
· Staffing an AI transformation
