People ask where the model lives. The more useful question is what else is
living next to it.
An on-premise AI node runs six things, and only one of them is a model.
When someone pictures on-premise AI they usually picture a model file sitting on
a server. That picture is not wrong, it is just the least interesting sixth of
what is actually there — and the other five-sixths are where deployments succeed
or fail.
Here is the full inventory.
1. The inference server
vLLM, holding model weights in GPU memory and serving requests over an
HTTP API.
This is the process that looks most like what people imagine. It loads
open-weight models, keeps them resident in GPU memory, and answers requests. It
speaks an OpenAI-compatible API, which matters more than it sounds: application
code written against a cloud API mostly does not need rewriting, it needs
repointing.
What it holds: model weights, and the attention cache for in-flight
requests. Weights are large and static; the cache is transient and is the thing
that actually constrains how many requests can run at once.
2. The model registry
A mapping from role to endpoint, with versions pinned.
Not a model. A lookup table with opinions. summarise-legal points at one
model, classify-ticket at a smaller and faster one, embed at an embedding
model. Application code asks for a role and never names a model.
What it holds: the mapping, the pinned versions, and a change log.
Why it exists: so that upgrading a model is a one-line change instead of a
migration across every application that calls it, and so that "which model
answered this question in March" is a question with an answer.
3. The vector index and the lexical index
Two indexes over the same corpus, because two kinds of question need two
kinds of search.
The vector index holds embeddings — numerical representations of meaning — so
that a question about indemnity can find a clause that says hold harmless.
The lexical index holds terms, so that a request for policy AB-4471 returns
AB-4471 and not AB-4417.
What they hold: for the vector index, one embedding per chunk plus the chunk
text and its metadata. For the lexical index, the terms and their field
weightings. Neither holds your original documents — those stay where they are.
Sizing note: the index is usually smaller than people expect, and the
original corpus is usually larger. Plan disk for the corpus you will re-parse,
not for the index you will query.
4. The ingestion pipeline
A background process that turns documents into indexed chunks, and it is not
on the request path.
It collects from file shares, document management systems, mailboxes and
ticketing systems; parses across thirty-plus formats with layout awareness;
chunks; embeds; and writes to both indexes. After the first pass it runs
incrementally, picking up what changed.
What it holds: the parse state, and — the part that matters most — the
entitlement mapping. Which people can see which documents, captured from the
source system at ingestion so it can be enforced at retrieval.
That mapping is the single most important thing on the box, and it is the one
nobody pictures. An index that does not inherit the source estate's access model
will happily surface a document to someone who could not open it in the system
it came from.
5. The agent runtime
The process that does multi-step work, under budgets.
Agents are defined in YAML — a role, a set of tools, a data scope, a step and
token budget. The runtime executes them, calls tools, enforces the budgets, and
trips a circuit breaker when a trajectory starts repeating itself.
What it holds: agent definitions, in-flight trajectories, and the scope
configuration that determines what each agent can reach.
6. The controls
Four things sitting on the request path, and one thing recording it.
A privacy filter before the model. An injection scanner between retrieved
content and the model. An output validator before anything is returned. And a
hash-chained audit trail recording all of it.
user / channel → privacy filter → injection scanner → local model
→ output validator → audit chain
What they hold: the filter's detection configuration and pseudonymisation
keys, the scanner's rules, and the audit chain — which is append-only and grows
for as long as your retention policy says it should.
So where does the application logic live?
Usually not on the box.
This trips people up. The AI node is infrastructure: it serves inference,
retrieves, runs agents and enforces controls. Your applications — the chat
interface, the internal tool, the integration into an existing system — live
where your applications live, and call the node over its API.
That separation is deliberate. It means the AI infrastructure can be upgraded,
resized or replaced without touching the applications, and it means a team that
wants to build something against it does not need to understand any of the
above.
What it looks like at rest
At the entry tier, all six of these run on one desk-side machine drawing about
240 W. At the mid tier they are distributed across a handful of nodes with the
registry routing between them. The processes do not change; their placement
does.
That is worth stating plainly because the mental model people bring — a rack, a
team, a project — comes from training infrastructure. Serving is a different
shape of problem, and it is a much smaller one.
The one-line version
If you take one thing from this: the model is the replaceable part. The
index, the entitlement mapping and the audit chain took real work to build and
they carry forward across every model you will ever run.
For the full picture — what physically arrives, who operates it, and what
happens when something breaks — see The anatomy of an on-premise AI
deployment.
