The anatomy of an on-premise AI deployment

On-premise AI runs the entire inference pipeline inside your building: open-weight models served on GPUs you own, grounded in your own documents by a local retrieval index, with no network path to any external model provider.

Most people evaluating this have a clear mental model of a cloud API and none at all of the on-premise equivalent. They are not sceptical — they are trying to picture it.

The hardware is smaller than expected

A departmental deployment serving fifty users is a single desk-side unit drawing about 240 W — less than a kettle. A mid-enterprise fleet serving several hundred staff is a few nodes in a normal rack on normal building power. Only datacentre-scale deployments need facilities engineering, and almost nobody starts there.

The reason is that inference is not training. Training a frontier model needs thousands of accelerators for months. Serving one to your staff needs enough memory to hold the weights and enough bandwidth to stream tokens at reading speed.

India landed cost, Q3 2026. Hardware pricing moves quarterly — the Spark figure alone moved 57% inside one quarter. Treat any published capex as a snapshot.

What runs on it

Five things: vLLM serving open-weight models with continuous batching and paged attention; a model registry mapping role to endpoint with pinned versions; a retrieval system holding a local hybrid index over your parsed documents, with five layers of memory around it; an agent layer running budgeted ReAct loops under per-project scope isolation; and a set of controls every request passes through.

The hard part is permissions, not hardware

Your document estate already has an access-control model, and the index must inherit it. Build an index without mirroring those permissions and you have built a retrieval system that will surface a document to someone who could not open it in the source system. Nothing malfunctioned — the entitlement was never carried across.

This is the single most common way an otherwise well-built deployment leaks, and it is invisible in testing, because the people running the pilot usually have broad access. The failure appears when someone with narrow access asks a question.

Permissions are therefore captured at ingestion and enforced at retrieval, so a user's candidate set is filtered by entitlement before anything reaches a model — in the retrieval layer, not in an agent's instructions. A constraint expressed in a prompt is a suggestion to a language model.

Where a source system's permission model is inconsistent, a person has to resolve it before ingestion. A deployment is a very effective way of discovering that a shared drive has been world-readable for six years.

What happens when someone asks a question

user / channel → privacy filter → injection scanner → local model
              → output validator → audit chain

The privacy filter screens the request before any model sees it, detecting Aadhaar, PAN and mobile numbers alongside international identifiers and pseudonymising with HMAC. Retrieval runs against the index filtered by that user's entitlements. The injection scanner inspects retrieved content, because retrieved documents are untrusted input. A local model selected by role answers. The output validator checks the response. Every step is hash-chained.

The failure mode is refusal. If a control is unavailable the request is refused; if the index is unavailable the request fails rather than being answered ungrounded. Fail-closed applies to grounding, not only to egress. It is easy to test during acceptance: break something on purpose and confirm the system says no.

Who runs it

It needs an owner with infrastructure responsibility, not a research team — and if nobody will own it, do not buy it. The recurring work is monitoring nodes, watching ingestion lag, reviewing the audit trail, and handling upgrades on a controlled schedule. Closer to running a database than to running a research project. Nobody is training models.

We state the disqualifier during scoping rather than after delivery. An unattended deployment drifts: ingestion falls behind, models go stale, and the audit trail becomes a record nobody reads.

What happens when a better model ships

The model is the most replaceable component in the system. Adopting a better open-weight model is a registry change plus a revalidation pass — the retrieval layer, the agents, the entitlement mapping and the audit chain are unaffected, because none of them depends on which model answered.

The durable assets are the parsed corpus, the index, the entitlement mapping and the accumulated audit record. Those compound. Buyers ask what happens if the model they run becomes obsolete; the better question is what happens to everything around it, and the answer is that it carries forward.

What a deployment does not include

Deciding which workloads are worth automating. Repairing source-document quality. Resolving an inconsistent permissions model. And change management — getting people to use the sanctioned tool instead of the one on their phone is the difference between a deployment and a shelf.

And the volume test: below roughly 90,000 queries a month, no on-premise deployment we build pays back inside three years, including the entry tier. If your volume is below that and your data is not sensitive, buy cloud.

What we will not tell you

No throughput figures — tokens per second, concurrent users — because we have no benchmark and will not invent one. No day-count schedule, because deployment duration depends on your corpus and your permissions model, and a vendor quoting a schedule before seeing either is quoting a hope.

Download the full paper — the three tiers in detail, the ingestion pipeline, the failure table, the six-phase deployment sequence, and a readiness checklist to use before you talk to anyone.

---

Companion piece: Multi-agent architecture for unstructured business data — why the architecture is shaped this way.