The work enterprises actually need from AI — assistants grounded in their own CRM, tickets, drawings, and records — does not require frontier-scale models, because it is a retrieval and orchestration problem, not a model-scale problem. That one sentence is the entire technical bet behind on-premise enterprise RAG, and this post is its defence: the mechanism, the measured numbers, and the honest caveat.
The insight: what enterprise AI work actually is
Look closely at what staff in a hospital, a bank, or an engineering firm actually ask an AI system to do: answer from our documents, draft from this CRM record, route this workflow, summarise that contract. None of it rewards a model for having memorised the public internet. All of it rewards a system for retrieving the right internal facts and reasoning over them carefully.
When retrieval supplies the facts — layout-aware chunking, hybrid keyword + semantic search, reranking, citations — and tools supply the actions, the model's job shrinks to reasoning over what it's given. That job fits a well-served 12B–35B open model, running on a 240 W node with 128 GB of unified memory, doing the work cloud APIs are sold for.
And it stays fresh without retraining: update a document in the morning, and answers reflect it by afternoon, because knowledge lives in the retrieval layer, not in the model weights.
This single observation is what lets a desk-size, 128 GB unified-memory mini-PC replace an eight-GPU server. It is also why the retrieval layer — not the model — is where answer quality is actually won.
Measured on our own cluster — not vendor datasheets
Every number below is measured on BiltIQ's own deployed cluster: live telemetry, not datasheet extrapolation.
| What we measure | Value | Why it decides the sale |
|---|---|---|
| Throughput on a 35B model | 29–31 tok/s | Interactive, not batch — the test a buyer applies in the first ten minutes |
| Speculative-decoding draft acceptance | ~88% | The single largest speed multiplier; why the small box feels big |
| Usable context demonstrated | 131K tokens | Whole contracts and case files fit in one pass |
| Fleet model memory | ~500 GB | Every model role resident at once — no swap-in latency |
| Power per node · whole-fleet typical draw | 240 W · ~300–500 W | Wall socket, office AC, no raised floor, no chilled water |
| Document format parsers · agent tools | 30+ · 30+ | Ingests what the customer actually has; acts through their own systems |
Under stress, the small box holds. 25 concurrent users on a single 240 W DGX Spark node — complex prompts, KV-cache compression active, zero errors. 929 tok/s aggregate on one RTX PRO 6000 workstation running 8 concurrent requests against the 35B reasoning model — measured live, no simulation.
Engineered at inference time, not by buying bigger hardware
The performance above doesn't come from exotic hardware; it comes from squeezing a fixed power budget with inference-time engineering.
Speculative decoding. A small draft model proposes tokens; the 35B verifies them in batches. At ~88% draft acceptance, this is the single biggest speed multiplier on the fleet — the reason a 240 W node sustains 29–31 tok/s on a 35B model and feels interactive rather than batch.
KV-cache compression + prefix caching. Our ManthanQuant compression achieves 5.12× compression at 0.983 fidelity, which — combined with expert offloading — pushes a 128 GB box from its 32K standard context to 131K tokens demonstrated in testing. Prefix caching makes repeated RAG context nearly free: the retrieved passages your staff query against all day cost almost nothing to re-serve.
MoE expert offloading. Mixture-of-experts models larger than a single node's GPU budget still serve interactively, because inactive experts are held in unified memory and paged in as routing demands.
Right-sized models, routed automatically
A production RAG system is not one model; it is a registry of roles. The model registry maps every endpoint by role, so agents route each task to the right-sized model automatically — no query pays for more model than it needs:
| Role | Model class | Why it's enough |
|---|---|---|
| General generation | 26B | Grounded drafting, tool calling, voice — the everyday workhorse |
| Deep reasoning | 35B MoE · 32K ctx (131K tested) | Long-document reasoning on one 240 W node |
| Multimodal | 12B | Streaming ASR, voice, and multimodal input |
| Medical | 4B specialist | Small + specialised beats big + generic |
| Embedding & rerank | 2B-class | The retrieval layer — where answer quality is actually won |
Note the last row. The models that most determine whether an answer is right — the embedding and reranking models — are the smallest in the fleet. Answer quality in enterprise RAG is won by retrieval precision, not generation scale.
The software that makes it a system, not a demo
Manthan — retrieval and memory. Layout-aware chunking across 30+ document formats (the contracts, scans, and spreadsheets a customer actually has, not clean markdown); hybrid keyword + semantic retrieval re-scored by a multimodal reranker so every answer carries citations; five-layer memory ending in an immutable audit trail. Citations are not a nicety here — for a bank facing an inspector or a hospital facing an auditor, an uncited answer is an unusable answer.
Agent OS — the workforce. Teams of AI specialists running in ReAct loops under iteration, token, and wall-clock budgets; 30+ tools plus MCP; CRM and ERP connectors; a DAG workflow engine with cron and event triggers; signed agent-to-agent delegation. The budgets matter operationally: an agent that can loop forever on a fixed-cost fleet wastes capacity, so every loop is bounded.
Privacy — the rail every call crosses. PII/PHI detection extended for Aadhaar and PAN; pseudonymisation so originals never reach a model; DPDP, GDPR, HIPAA, and CCPA validators; a tamper-evident audit hash chain. Every model call passes through it — the rail is architectural, not optional.
The honest caveat, stated plainly
The whole bet is that a well-served 12B–35B open model plus retrieval does the enterprise job. If frontier capability pulls decisively away on the specific tasks enterprise buyers care about — grounded answering, document reasoning, tool use — the value proposition narrows to sovereignty alone. We publish this risk in our own investor materials, and we watch the gap constantly. Today, for grounded enterprise work, the evidence — including our own measured deployments — favours the small box. That could change, and a vendor who won't say so isn't measuring.
What to take from this if you're evaluating on-premise RAG
Three tests worth applying to any on-premise RAG proposal, ours included. First, ask for measured throughput on the vendor's actual deployment hardware, not datasheet numbers — interactive tok/s under concurrency is the figure that decides whether staff use the system. Second, ask where citations come from — a RAG system that can't show its retrieval sources per answer will fail its first audit. Third, ask what happens to a document updated this morning — if the answer involves retraining, it isn't RAG, it's an expensive snapshot.
BiltIQ's Manthan retrieval engine and Agent OS run entirely on your premises — right-sized open models, cited answers, and a privacy rail on every call. Book a consultation: [email protected] · +91 89868 60088 · www.biltiq.ai
Frequently asked questions
Do enterprise AI workloads really not need frontier-scale models?
Grounded enterprise work — answering from internal documents, drafting from records, routing workflows — is a retrieval and orchestration problem that a well-served 12B–35B open model handles when retrieval supplies the facts and tools supply the actions. The model reasons over retrieved context rather than relying on memorised knowledge, which is why retrieval precision, not model scale, determines answer quality.
What throughput can an on-premise RAG deployment actually sustain?
Measured on BiltIQ's own cluster: 29–31 tokens/second sustained on a 35B model per node, 25 concurrent users with zero errors on a single 240 W DGX Spark, and 929 tok/s aggregate on one RTX PRO 6000 workstation at 8 concurrent requests. These are live-telemetry figures from deployed hardware, not vendor datasheet projections.
How does a 128 GB node handle 131K-token contexts?
Through inference-time engineering rather than bigger hardware: ManthanQuant KV-cache compression at 5.12× with 0.983 fidelity, plus MoE expert offloading, extends usable context from the 32K standard to 131K demonstrated in testing. Prefix caching additionally makes repeated RAG context nearly free to re-serve.
How do answers stay current without retraining the model?
Knowledge lives in the retrieval layer, not the model weights, so updating a document updates the answers — a document changed in the morning is reflected in responses by afternoon. This is the core operational advantage of RAG over fine-tuning for fast-moving internal knowledge.
Does every answer really carry citations?
Yes — Manthan's hybrid retrieval is re-scored by a multimodal reranker so every answer is grounded in identified source passages, backed by five-layer memory ending in an immutable audit trail. For regulated buyers, citation and auditability are requirements, not features.
What document formats can the system ingest?
Manthan parses 30+ formats with layout-aware chunking — built for the contracts, scans, drawings, and spreadsheets enterprises actually hold, rather than assuming clean text. The agent layer separately exposes 30+ tools plus MCP for acting through the customer's own systems.
