Almost every conversation about on-premise AI is a conversation about GPUs. It
is the wrong conversation.
The reason a modest node serves a department at interactive speed is
inference-time engineering, not the accelerator underneath it.
Buy the hardware, load the weights, serve the requests — that is roughly how
people imagine an on-premise deployment works, and it is why the first question
is almost always about the GPU. But between a request arriving and the first
token leaving, there is a body of engineering that determines whether the same
hardware serves five people or five hundred. Comparing accelerators on raw
specifications predicts very little about what a deployment will actually
serve.
This post explains that work in plain terms. Not because it is our secret, but
because almost nobody in this market writes about it, and a buyer who does not
know it exists will make the decision on the wrong axis.
Start with why naive serving is so bad
A naive model server loads one model, answers one request, and idles.
A GPU is a machine for doing enormous numbers of arithmetic operations at once.
A single request from a single user does not come close to filling it. So the
accelerator you paid for spends most of its life waiting — and worse, the memory
it holds is reserved as though the work were happening.
Everything below is a response to that one fact.
Continuous batching
Requests arriving at different moments are folded into the same batch, and
finished requests leave the batch without waiting for their neighbours.
The obvious fix to an idle GPU is to process several requests at once. The naive
version of that is static batching — wait, collect a group, run them together.
It fails badly in practice, because generation lengths differ wildly: one
request wants twelve tokens and another wants two thousand, and the whole batch
waits for the slowest.
Continuous batching lets requests join and leave a running batch. A short
request finishes and departs; its slot is immediately filled by whoever is
waiting. The GPU stays busy, and a user asking a small question does not queue
behind someone generating a report.
This is what our serving layer does, and it is the single largest factor in
the economics.
Paged attention
Attention memory is managed in fixed-size pages, the way an operating system
manages RAM.
During generation a model keeps a cache of its attention state — the KV cache —
and that cache grows with every token. The naive approach reserves a contiguous
block sized for the longest output the request might produce. Most requests
never use most of it, so a large fraction of expensive GPU memory sits reserved
and empty. Memory, not compute, becomes the limit on concurrency.
Paged attention allocates that cache in small pages, on demand, and they need
not be contiguous. The effect is the same as virtual memory in an operating
system: far less waste, far more concurrent requests on the same card.
This is also what our serving layer does. Between these two techniques, the
question "how many people can this node serve" stops being a question about the
GPU and starts being a question about how well the memory is managed.
The wider toolkit
Beyond those two, the field has developed a set of techniques that all attack
the same problem from different directions. We are describing them here as the
state of the art, not as a feature list — where a technique is not in our
serving layer, we are not going to imply that it is.
Quantisation stores weights at lower numerical precision — eight bits, or
four, instead of sixteen. The model gets smaller and faster; some quality is
traded away. How much depends heavily on the model and the task, which is why
quantisation is a decision to be measured on your workload rather than a setting
to be switched on.
KV-cache compression attacks the same memory problem paged attention
attacks, from the other side: rather than allocating the cache more efficiently,
make the cache itself smaller.
Speculative decoding uses a small fast model to guess several tokens ahead,
then has the large model verify the guess in one pass. When the guess is right —
which it often is, because much of language is predictable — several tokens are
produced for roughly the cost of one.
Prefix caching notices that many requests share an opening. A long system
prompt, a standard instruction block, a retrieved document reused across
questions — computed once, reused rather than recomputed. In a retrieval-heavy
workload, where every request carries retrieved passages, this matters more than
it does in a chat product.
Expert offloading applies to mixture-of-experts models, which activate only
a fraction of their parameters for any given token. That makes it possible to
keep the rarely-used parts somewhere cheaper than GPU memory and fetch them when
needed — trading a little latency for a much larger model on the same hardware.
Why this decides the buying question
Every one of those techniques converts into either more users on the same
hardware or the same users on less hardware. That is the entire economic
argument for owning infrastructure, and it is invisible in a specification
sheet.
It also explains something that puzzles people about the crossover maths.
On-premise economics improve with sustained volume, because the fixed cost is
already paid and the marginal cost of the next request is electricity. But that
is only true if the accelerator is actually being used. A node running at ten
percent utilisation is an expensive way to be slow. The techniques above are what
close the gap between the hardware you bought and the hardware you are using.
And it explains why long contexts change the answer so sharply. Retrieval-heavy
and agentic workloads carry large amounts of retrieved material in every
request. Memory management and cache reuse stop being optimisations and become
the constraint — which is precisely the workload an enterprise deployment runs
by construction.
What we will not tell you
No numbers. Not tokens per second, not time to first token, not concurrent
users.
We have no published benchmark, and a figure produced on our hardware with our
models on our documents would tell you nothing reliable about yours. Serving
performance is a property of a specific model at a specific quantisation on
specific hardware under a specific request mix. Any vendor quoting you a single
throughput number without naming all four is quoting a number that cannot be
checked — which is the same as quoting no number at all.
What we will do instead is run it on your workload during a pilot and show you
what it does.
The question to ask a vendor
Not which GPU. Ask:
"What is your serving layer, and what does it do about memory?"
If the answer is a model name and a card name, the vendor is selling you
hardware. If the answer describes batching and cache management, they have
thought about the part that determines what you actually get.
