Small Language Model (SLM)

What is a small language model?

A small language model, or SLM, is a language model compact enough to run on hardware you already control: a single GPU, a business laptop, sometimes a phone. It does the same everyday text work as a bigger model, summarising, answering questions, sorting email, pulling fields out of documents, but it fits where a large model cannot.

The contrast is with a Large Language Model (LLM), which can carry hundreds of billions of parameters and needs a rack of datacenter GPUs to run. An SLM runs on one machine, and the smaller ones run on a device with no server behind them at all.

There is no crisp line between the two. A common rule of thumb puts an SLM under roughly 10 billion parameters, and Microsoft uses that figure in its own documentation, yet it still files its 14-billion-parameter Phi model on the large side. The number is soft on purpose. In practice "small" describes what the model fits on, not a fixed parameter count.

Why teams reach for a smaller model

The reasons are rarely about raw quality. The first is cost. An LLM is usually rented through an API that bills per token, so the meter climbs with every request. A self-hosted SLM runs at a fixed cost, because you pay for the server rather than for each call. For a job that runs inference thousands of times a day, that is the gap between a bill that moves with volume and one you can predict.

The second reason is latency. Fewer parameters mean less compute per token, so answers come back faster and one machine handles more requests at once. The third is data residency. A small model can run entirely inside your own network, so the prompt and the response never leave the building. For customer records, staff files, or anything a regulator asks about, that is often the deciding factor. The fourth is offline operation. A model on the device keeps working with no connection: a laptop on a shop floor, a handheld in a warehouse, a phone in the field.

The trade you are making

A smaller model knows less about the world and reasons less well on open-ended problems. Ask it something broad, hand it messy or ambiguous input, or expect several steps of reasoning, and a large model still wins. The honest framing is that you give up general breadth for a model that is cheaper, faster, and yours to run.

For a narrow task, that breadth is often something you were never going to use, and two techniques close most of the gap. Fine-tuning trains the model on your own examples so it learns the shape of your specific task, and a fine-tuned small model can match a much larger one on that single job. Grounding it with retrieval, the pattern behind RAG, feeds the model the right documents at answer time so it works from your data instead of its own memory. Do both and a small model stops competing on world knowledge and starts competing on your task, where it does not need to be big.

How a model gets small

Two methods do most of the work, and they are separate ideas. Model distillation trains a small student model on the outputs of a large teacher, so the student inherits much of the behaviour without the size. Quantisation shrinks a model after training by storing each weight in lower precision. Weights usually sit in 16-bit numbers; quantisation rounds them to 8-bit or 4-bit integers, which cuts the memory the model needs to load and run.

The arithmetic is easy to see. A 7-billion-parameter model at 16 bits per weight needs about 14 GB just to hold its weights, that is 7 billion times 2 bytes. Drop each weight to 4 bits and the same model needs about 3.5 GB, which is the difference between server-only and fitting on a laptop GPU. The two methods stack: you can distil a small model and then quantise it.

When a small model is the right call

The decision rule is short. A narrow, well-defined task with good context rarely needs a frontier model. If you can describe the job in a sentence and hand the model the data it needs, size stops being the thing that decides the outcome.

Tasks that fit that shape well:

• Classifying and extracting. Sorting incoming email to the right team, recognising invoices, pulling amounts and dates out of documents.

• Structured lookups. Turning a plain-language question into a query with NL2SQL, where the database schema already gives the model the context it needs.

• Sensitive or regulated work. Anything that cannot leave your environment: run it locally and the compliance conversation gets a lot shorter.

• High volume at a fixed price. Hundreds of thousands of small, repetitive calls a month, where a per-token bill would hurt.

Test the edge cases before you commit. A small model is strong on the standard case and trips sooner on the unusual one: a document in an odd format, a question just outside its domain. Keep a fixed set of real examples with their expected answers, so you can tell whether a task genuinely needs the bigger model or whether the small one already does the job.