"Is it cheaper to run our own?" is the wrong first question for most sovereign AI projects, because the teams who most need private deployment do not have a choice — a contract or a regulation has already made it for them. But it is always the second question, and it deserves an honest answer rather than a sales one.
The short version: self-hosting replaces a variable per-token bill with a fixed capital cost plus a recurring operational one. Whether that is cheaper depends almost entirely on utilisation.
The cost structure, in full
1. Hardware
The obvious line, and the one most estimates stop at. GPU memory requirements are covered in detail in what GPUs you need to self-host an LLM, but the summary is that model size, concurrency and context length together drive the requirement, and concurrency is frequently the dominant term.
Two options, and the choice is mostly about what your constraint really is:
- Buy when inference must stay on-premises or air-gapped, or when utilisation will be high and steady.
- Your own cloud account, in a chosen region, when the requirement is jurisdictional rather than physical. Far less procurement, and it satisfies most residency rules.
Hardware is also not the whole capital cost. Servers need somewhere to live, with power and cooling adequate to run GPUs at sustained load. In an existing data centre this is marginal; in an office it may not be.
2. Power and facilities
GPUs draw meaningful power under load and reject meaningful heat. For an on-premises deployment this is a real recurring line, and it scales with utilisation — which means the same factor that improves your cost per request also raises this bill. Worth a conversation with whoever runs your facilities before you commit.
3. Engineering time
Consistently the most underestimated line, and frequently larger than the hardware.
A self-hosted deployment needs someone to serve the model, monitor it, handle updates, and respond when it breaks. That is a real ongoing commitment, not a one-off setup task. Teams with existing platform or MLOps capability absorb it easily. Teams without it are acquiring a new operational responsibility, and it is worth pricing honestly rather than assuming it will be free.
This is precisely the gap managed AI services exist to fill — the same isolated environment, run by someone whose job it is.
4. Building the system around the model
Serving a model is not the project. A useful assistant needs ingestion, retrieval, evaluation, access control and audit logging built and maintained around it. That work exists whether you self-host or call an API, but it is often mentally filed under "the model" and therefore missing from the comparison.
5. Updates and retraining
Models improve. Your documents change. Budget for periodically re-evaluating against a newer model and re-indexing as content grows. Neither is expensive individually; both are invisible in a first-year estimate that assumed the system was finished at launch.
The comparison that actually matters
Per-token API pricing has a property that makes it deceptively attractive early and deceptively expensive late: it scales with use, forever.
Self-hosting has the mirror-image property: a large cost before you process a single request, and a low marginal cost after.
This gives the familiar shape:
| Per-token API | Self-hosted | |
|---|---|---|
| Up-front cost | Near zero | Hardware, setup, integration |
| Cost per request | Fixed rate, forever | Approaches marginal power cost |
| Cost at low volume | Cheap | Poor value |
| Cost at high volume | Grows without limit | Flat |
| Cost predictability | Varies with usage | Predictable |
| Idle cost | Zero | Full |
That last row is the one teams miss. An idle GPU costs the same as a busy one. A self-hosted deployment used by twelve people for an hour a day is expensive per useful request, however cheap the hardware looked.
Which produces the actual rule:
Self-hosting gets cheaper the more you use it. If usage will be low or sporadic, the economics favour an API — and if an API is not permitted, then cost was never the deciding factor and the comparison is academic.
Two things that change the arithmetic
Per-seat pricing scales with headcount, not value. Where a hosted tool is licensed per user — coding assistants are the clearest example — the bill grows every time you hire, regardless of how much each person uses it. Your own hardware does not care how many people are logged in, only how many are querying at once. For larger engineering teams this shifts the break-even substantially, which is part of the case for a private coding assistant.
A smaller model changes everything downstream. Serving cost is paid on every request. A task-specialised smaller model that matches a large general one on your specific job reduces hardware, power and latency simultaneously. This is where fine-tuning can pay for itself outright rather than merely improving quality — training is paid once, inference is paid always.
What you get that does not appear on either side
Some of the value of self-hosting is real but does not show up as a number in a spreadsheet, and it is worth naming rather than smuggling in:
- Cost predictability. A fixed bill you can plan against, rather than one that tracks adoption.
- No vendor dependency. Nobody deprecates your model, changes the terms, or has an outage on your behalf.
- Data that never leaves. For many organisations this is the entire reason, and it is not a cost saving.
- Artifacts you keep. Weights, pipeline and index are yours, and they keep working whether or not the relationship continues.
Conversely, the honest debit column: you have taken on an operational responsibility, and you no longer get model improvements for free.
How to get a real number
Generic estimates are worth very little here, because the answer swings by an order of magnitude on utilisation. What produces a defensible figure:
- Estimate real volume — requests per day at peak, not aspirational adoption.
- Size the hardware for that concurrency and your latency target.
- Price the same volume at current API rates for a comparable model.
- Add engineering time to the self-hosted side, honestly.
- Compare over three years, not one. Hardware is capital; a per-token bill is not.
If the two come out close, the non-financial column above is what decides it — and usually it decides it firmly in one direction, because the teams doing this exercise mostly have a constraint that made the decision before the spreadsheet did.
How AxcelerateAI Helps
We size private deployments before clients commit to hardware:
- Workload sizing from real concurrency and latency targets.
- Deployment on your infrastructure, on-premises, air-gapped or in your own cloud account.
- Managed operation where you would rather not take on the running of it.
