Inference

What is inference?

Inference is the moment a trained AI model is used. You send new input to the model, and it calculates an output: a prediction, label, answer, translation, image, or next token.

A chatbot answering a customer question is running inference. So is an invoice model assigning a category, a vision model detecting damage in a photo, or a recommendation model ranking products.

Training is the work that creates or adapts the model. Inference is the repeated work the model does after that.

Training versus inference

Training
During training, the model learns patterns from examples. Its internal weights change. This is expensive, usually done on large GPU clusters, and often handled by the model provider.

Fine-tuning
Fine-tuning continues training an existing model on a narrower set of examples so it behaves better for a task, format, domain, or tone.

Inference
During inference, the weights are fixed. The model applies what it already learned to the new input. A language model reads the prompt, calculates the next token, appends it, and repeats until the answer is done.

For many business teams, inference is the AI cost they actually see. They use existing models through an API, so they are not paying to train the base model. They pay every time the model is called.

What drives inference cost?

For language models, cloud APIs usually charge by tokens. Input tokens are the text you send. Output tokens are the text the model generates. Output is often more expensive because the model produces it step by step.

The cost grows with three things:

• Prompt size. Long instructions, conversation history, documents, and retrieved context all add input tokens.

• Answer length. Longer output means more generated tokens.

• Call volume. A workflow that calls a model ten times per ticket is ten model calls, not one.

Cost controls exist. Batch processing is cheaper on some platforms for jobs that can wait. Prompt caching can reduce the cost and latency of repeated context. Smaller models can handle simple classification or extraction work at a lower price than flagship models.

Where can inference run?

Cloud API
You call a provider such as OpenAI, Anthropic, Google, Microsoft, or AWS and pay per usage. This is the easiest route: no model serving, no GPU management, recent models available quickly. The trade-off is that your data is processed by the provider or cloud platform, so residency, logging, retention, and contracts matter.

Your own servers
You host an open or licensed model on your own infrastructure or cloud GPU instances, often with serving software such as vLLM, NVIDIA Triton, or Text Generation Inference. This gives more control over data, model choice, and runtime. It also gives you the operational work: capacity planning, updates, monitoring, security, and cold starts if the service scales down.

On device
Small models can run on a laptop, phone, browser, or industrial device. That can keep data local and work offline. The limitation is model size and throughput. A phone can help one user; it is not a shared inference server for a whole company.

Latency and throughput

Two numbers decide whether inference fits a process.

Latency is how long one request takes. For a chatbot, users notice the time to first token: how quickly the answer starts streaming.

Throughput is how much work the system handles overall: tokens per second, requests per minute, images per hour, or documents per night.

These goals can pull in different directions. Batching improves GPU utilisation and total throughput, but it may make an individual user wait longer. A customer-service chat needs low latency. A nightly document classification job can trade latency for throughput and lower cost.

What to watch out for

Long prompts become recurring cost
A giant system prompt or full policy manual in every request may be acceptable for a prototype, but painful at production volume.

Data location still matters
If the prompt contains personal data or confidential business data, inference is data processing. Check the region, provider terms, retention settings, and processor agreements.

Model upgrades change behaviour
A provider-side model update can change outputs, latency, or price. Keep evals around important workflows.

Self-hosting is not automatically cheaper
A GPU server costs money while idle. Self-hosting usually pays off only with predictable volume, strong data-control requirements, or a model that fits your workload well.

Output limits protect budgets
Set max tokens, timeouts, and retry limits. A stuck agent loop can multiply inference calls quickly.