For plenty of engineering organisations, the coding assistant conversation ends before it begins. Client contracts prohibit source leaving the network, the work happens on an isolated network with no outbound route, or the codebase contains credentials and customer data that nobody is willing to see transmitted on every keystroke.
None of that removes the productivity argument. It just means the assistant has to run on your side of the perimeter. This is what that involves.
What "self-hosted" has to mean
The term is used loosely, and the distinctions matter when a security review starts asking questions.
- Self-hosted inference. The model runs on your hardware. No code leaves for completion or chat.
- Self-hosted indexing. Your repositories are embedded and indexed on your hardware too. Easy to overlook, and a pipeline that serves the model locally while calling a hosted embedding API is shipping your entire codebase out a different door.
- No licence call-home. Some commercial tools that advertise on-premises deployment still contact a licence server. On a genuinely air-gapped network this disqualifies them, whatever the marketing says.
- Self-hosted logging. Prompt and response telemetry sent to a vendor's monitoring service carries code fragments with it.
The test that settles all four: disconnect the network and see whether it still works. It is a cheap test and it is remarkably clarifying.
The three workloads, which are not the same problem
Teams tend to say "coding assistant" and mean one of three things with very different engineering profiles.
Inline completion
Suggestions as you type. The distinguishing constraint is latency: this has to return in a few hundred milliseconds or developers turn it off, and they will not turn it back on. That makes it the hardest of the three to serve well, and it argues strongly for a smaller, faster model rather than the largest one that fits.
Dedicated code models matter here in a specific way: they are trained for fill-in-the-middle, where the model sees code on both sides of the cursor. General-purpose models are usually poor at this regardless of their overall quality.
Codebase questions
Why does this module do that? Where is this rule enforced? What breaks if I change this? This is retrieval over your repositories, internal documentation and decision records — architecturally the same system as a private knowledge assistant, with code as the corpus.
Latency tolerance is much higher: a few seconds is fine for a considered answer. This is usually where the value is largest and the engineering risk lowest, which makes it the sensible thing to build first.
Review and change assistance
Summarising a diff, flagging patterns your team has decided against, pointing at the internal convention a change contradicts. Batch-shaped, latency-tolerant, and dependent on the assistant knowing your conventions — which means retrieval over your own code, not general knowledge about how code is usually written.
Choosing what to serve
Three decisions, in this order:
1. Licence. Check what you are permitted to do with the weights before evaluating anything. Some strong code models carry terms that restrict commercial use or attach conditions on derivatives.
2. Hardware. Inline completion at low latency across a team of thirty is a concurrency problem as much as a model-size one. Work out VRAM against concurrent users rather than a single request; GPU requirements for a self-hosted LLM covers the arithmetic, and the KV-cache section applies with particular force here.
3. Languages. Open-weight code models have uneven coverage. Performance on mainstream languages is broadly good; performance on older stacks, in-house frameworks and less common languages varies enough that it must be tested rather than assumed. If your team works in something unusual, this is also where an internally trained model earns its cost, because your own repositories are the best available training data for your own conventions.
We cover the general selection process in which open-weight LLM should you self-host.
Connecting it to the editors people already use
The practical route is usually simple: serve the model behind an OpenAI-compatible endpoint on your internal network, and point editor extensions at that URL. Most open editor integrations accept a custom base URL, so developers keep the tools they know and you have exactly one service to secure.
That endpoint deserves the same treatment as any internal service holding sensitive data: authentication, TLS, network-level access restriction, and request logging. An unauthenticated model endpoint reachable across an internal network, with your codebase indexed behind it, is a significant exposure with a friendly interface.
Practical guidance on standing the endpoint up is in deploying an open-weight LLM with vLLM.
What breaks in an air-gapped install
Specifically, and in our experience repeatedly:
- Model and tokenizer downloads at start-up. Fetch everything in advance, install from local paths, and point at directories rather than remote identifiers.
- Editor extensions that check for updates. Some fail closed and stop working entirely rather than degrading.
- Container images pulled at deploy time. Mirror them internally.
- Telemetry that blocks on a failed send. Usually configurable, occasionally not.
- Licence servers. The one that most often ends an evaluation.
Validate all of this by disconnecting rather than by reading documentation. Vendors describing a product as supporting offline use often mean it tolerates intermittent connectivity, which is a different claim.
Measuring whether it is worth it
Adoption is easy to mistake for value. Agree what to measure before the build:
- Suggestion acceptance rate, for completion.
- Time to a useful answer on codebase questions, against the honest baseline of asking a colleague.
- Correctness on your own code — a benchmark built from your repositories, since public coding benchmarks say nothing about how a model handles your internal framework.
- Latency at realistic concurrency, tracked continuously. This is the number that silently kills adoption, and it drifts as usage grows.
Without a baseline, a fine-tune or a model upgrade is a change rather than an improvement, and you will have no way to tell them apart.
Access control is not optional
Your repositories already have permissions. An assistant that indexes everything and answers anyone has flattened them.
Enforce permissions at retrieval time, so passages a user may not read are filtered out before they reach the model. Asking the model to respect access rules in a prompt is not a control, and it will not survive a security review.
How AxcelerateAI Helps
We build private coding assistants that run entirely inside client networks:
- Model selection and serving sized for your languages, team size and latency target.
- Retrieval over your repositories and internal documentation, with repository permissions carried through.
- Air-gapped installs with no runtime dependency on anything outside your network, plus handover so your team updates models themselves.
