Local LLM (on-premises inference)

What is a local LLM?

A local LLM is a language model that runs on hardware you control instead of on a provider's servers. The weights sit on your own disk, the model loads into your own memory, and the prompt never leaves the machine. That machine can be a workstation under a desk, a server in your own rack, or a virtual machine in a cloud account only your team can reach.

On-premises inference is the same idea with the emphasis on place. Inference is the part where a trained model answers a question, and on-premises says that answering happens on your infrastructure rather than on someone else's.

This is a deployment decision, not a model decision. You need an open-weight model to do it, because the weights of a closed model cannot be downloaded, but choosing open weights and hosting them yourself are two separate steps, and plenty of companies take the first and skip the second.

One thing local is not: calling a hosted API in a European region. Your text still leaves your network and an external party still processes it. That is often the right arrangement, but it puts a contract behind your data instead of a machine.

Why companies run a model on their own hardware

Five reasons come up in practice, and only one of them is about money.

• Data that is not allowed to leave. Patient records, personnel files, unpublished figures, material under professional secrecy. When the answer to "may this text go to an external processor" is no, running the model yourself removes the question rather than answering it.

• A bill that stops moving with volume. A hosted model bills per token, so the invoice tracks usage. Your own server costs the same whether it answers once a day or a million times.

• A model nobody can retire. Providers deprecate versions on their own schedule and give you notice, not a veto. A model on your disk behaves as it did the day you tested it, which matters for anything you have validated.

• Sites without a reliable connection. A production hall, a vessel, a handheld in a warehouse with patchy coverage. A model on the device keeps working when the line drops.

• Latency you can predict. No network hop and no queue shared with other customers, so response times stay in a narrow band instead of spiking when the provider is busy.

What it takes to run one

Four things, and the fourth is the one that gets forgotten.

An open-weight model with a licence you have read. Llama, Mistral, Gemma, Qwen and the gpt-oss family all publish downloadable weights, each with its own terms on commercial use and redistribution.

Enough memory. Memory is the binding constraint and the arithmetic is short. Weights are usually stored at 16 bits, so loading a model takes roughly two bytes per parameter: about 16 GB for an 8 billion parameter model, about 48 GB for a 24 billion one, before the memory the context itself needs. Quantisation stores each weight in fewer bits and cuts that hard: the common 4-bit build of Llama 3.1 8B is a file of about 5 GB, which fits inside a 16 GB graphics card. Mistral says its 24 billion parameter Mistral Small 3.1 needs about 55 GB of GPU memory at full precision, and that quantised it fits within a single RTX 4090 or a 32 GB MacBook. Three rungs, as of September 2026: 7 to 8 billion quantised runs on a consumer graphics card or a recent laptop, 20 to 30 billion needs one strong GPU, and 70 billion and up needs a serious workstation or server. NVIDIA sells its DGX Spark, a desk-side machine with 128 GB of unified memory, for fine-tuning up to 70 billion parameters and inference up to 200 billion. Past that you are into multiple GPUs and a rack.

A serving runtime. The program that loads the weights and answers requests. llama.cpp is built for inference with minimal setup, from a laptop CPU to a datacentre GPU, supports quantisation from 1.5 bits up to 8, and can split a model across CPU and GPU when it does not fit in video memory alone. Ollama wraps that in something a developer installs in a minute, on NVIDIA cards from compute capability 5.0 upwards, AMD through ROCm and Apple silicon through Metal. vLLM is what you reach for when many people hit the same model at once: it wants Linux and, on NVIDIA, a card of compute capability 7.5 or higher, and it is built for throughput rather than for a laptop.

Someone who owns it. Drivers, the runtime, the model files and the monitoring that tells you it stopped. Small system, but a production system, and it has no support line.

What it costs against a hosted API

Take a document workflow: 20,000 documents a month, roughly 2,000 tokens of input and 300 of output each. On Anthropic's price list of 4 September 2026 the small tier, Claude Haiku 4.5, costs 1 dollar per million input tokens and 5 per million output. That is 40 dollars of input and 30 of output, so about 70 dollars a month. The top tier on the same list, at 10 and 50 dollars, would be about 700.

Now the local side. A desk-side box of that class costs a few thousand dollars: the DGX Spark launched at 3,999 dollars. Over three years that is a bit above 100 dollars a month before anything else. Add power and the few hours a month somebody spends patching and watching it, and a realistic all-in figure lands between 300 and 500 dollars. Almost all of that is the person, not the hardware.

So at this volume the hosted small tier wins by a wide margin, and the crossover sits above 100,000 documents a month, where 70 dollars becomes 400. Against the top tier you are ahead on the invoice already, but a 24 billion parameter model on your own box is not doing what the top tier does, so that comparison only holds if your own testing says the smaller model is good enough. And an idle GPU costs what a busy one costs, so a machine at ten percent load has a cost per document ten times higher.

Local LLM versus hosted API: what is fixed and what is variable

The cleanest way to see the choice is to ask which parts are fixed and which parts move.

Cost. Local is fixed, hosted is variable. You buy the machine once and the bill barely moves with usage, which is comfortable at high steady volume and painful when volume is low. Hosted moves with every call, which is comfortable while you are still finding out whether the thing works at all.

Capability. Local is fixed, hosted moves, and hosted starts ahead. The strongest models are closed and reachable only through an API, so what you can download trails them by a few months. The model you downloaded then behaves the same in eighteen months, while the hosted one gets better over that period without you doing anything, and also changes underneath you, gets retired and gets repriced.

Responsibility. Local is all yours, hosted is shared. Uptime, patching, capacity and the behaviour of the raw model are your problem when you run it. With a hosted model those sit with the provider, along with a contract that says what happens when they fail.

Local buys you a fixed bill and a fixed model, and charges you for it in work.

The middle ground, and where to start

Local and a US hosted API are the two ends. Most of the useful answers sit between them.

A European processing region. Microsoft's Foundry deployment types include a Data Zone option where prompts and responses are processed only within the specified data zone, and the European zone follows the Azure EU Data Boundary. The narrower Standard and Regional Provisioned types keep processing inside the Azure geography you name.

Dedicated capacity instead of per-token billing. If you want a predictable bill rather than a private machine, providers sell reserved throughput. Azure bills provisioned throughput units hourly on the capacity you deploy rather than on the tokens you consume, with reservations for a term discount. Amazon Bedrock does the same with model units, hourly, with no commitment or a one-month or six-month term.

A sovereign cloud. AWS launched its European Sovereign Cloud on 15 January 2026, with a first region in Brandenburg that is physically and logically separate from other AWS regions, run through EU-incorporated companies and operated by EU residents.

Check that the middle ground exists for the model you want. Anthropic's own data residency controls, as of September 2026, let you pin inference to the United States or leave it global, with no EU-only value.

So for most SMEs the order is: start hosted, in a European region, with a processing agreement in place. Article 28(3) of the GDPR sets out what that agreement has to cover, from documented instructions and confidentiality through to sub-processors, deletion at the end and the right to audit. Move a workload local only when you can name the reason, a class of data that may not go out or a volume that makes the sum work, and then move one task rather than everything. A single classification or extraction job is a fair test of whether your team wants to own a model server. A general assistant for the whole company is not.

What to watch out for with a local LLM

Local does not mean safe. Your model has the same weakness as any other: it cannot tell instructions from data. A prompt injection hidden in a supplier PDF works just as well against your machine, and if that model can reach your file share the blast radius is larger rather than smaller. Access control, logging and the question of which tools the model may call all still apply.

You lose the vendor's tuning and their uptime. A hosted model arrives with safety behaviour, abuse handling and an availability commitment. A raw open-weight model arrives with none of it.

Nothing updates itself. The model server, its dependencies and the GPU driver all need patching, and a new model version is a project rather than a notification. Keep a test set of your own examples with the answers you expect, or you cannot tell whether an upgrade helped.

The GDPR does not disappear. Running the model yourself takes an external processor out of that step, so no processing agreement is needed for it. You are still the controller, the security obligations of Article 32 still apply to the machine, and the record of what people typed into the model is now personal data on your own disk.