Model serving

What is model serving?

Model serving is how a trained machine learning model becomes something other systems can actually call. You put the model behind an interface, send it new data, and get a prediction, score, or classification back. Training builds the model; serving is the part that runs it in production, day after day.

The prediction step itself is inference. Serving is the machinery around it: the server or job that loads the model, accepts input, runs inference, and hands the answer back to whatever asked.

A churn model might score every customer overnight and drop the results in a table your CRM reads the next morning. A fraud model might score each card transaction the instant it happens. Same idea, two very different serving setups, and picking between them is the first thing you decide.

Real-time serving versus batch scoring

The deciding question is whether a caller is waiting on the answer in the moment.

Real-time endpoints

A real-time endpoint is a web service that answers one request at a time over HTTP. A system sends one record, the endpoint returns one prediction, usually in tens of milliseconds. Azure Machine Learning calls these online endpoints; Databricks Model Serving exposes each model as a REST API. You get a latency budget to hit, autoscaling that adds or removes compute as traffic rises and falls, and cold starts: if the endpoint scaled down to zero while idle, the first request afterwards waits for compute to spin back up. Databricks notes this can take ten to twenty seconds, and sometimes minutes.

Batch scoring

Batch scoring runs on a schedule. A job wakes up, reads a pile of records, scores all of them at once, and writes the predictions to a table or file for something else to pick up later. There is no endpoint waiting on callers and no per-request latency to defend. Azure batch endpoints run these jobs asynchronously across a compute cluster and store the output to a data store. This is ordinary batch processing with a model in the middle.

Here is the honest part: most business predictions do not need an answer in the same second. A nightly batch that scores every customer and writes the results to a table covers churn scoring, segmentation, and demand forecasts. Yet plenty of teams build and run a real-time endpoint for a problem a nightly batch job would solve more cheaply and with far less to monitor. Reach for real-time when a decision has to happen inside a live request, like a fraud check or on-site personalisation, and default to batch for everything else.

Rolling out a new model safely

Replacing a live model is risky, so serving platforms borrow release patterns from software deployment. Three come up again and again:

• Shadow deployment. The new model receives a copy of live production traffic and its answers are discarded, so you compare it against the current model on genuine data without any customer seeing its output.

• Canary. You route a small slice of traffic, say ten percent, to the new model and raise its share only while it stays healthy, which limits the damage if something is wrong.

• A/B test. You split live traffic between two model versions and compare them on a business outcome rather than accuracy alone, to see which one performs better in production.

Features have to match between training and serving

A model learns from features, the input columns computed from raw data. If a feature is computed one way when the model is trained and a different way when it is served, predictions quietly go wrong. Google's Rules of Machine Learning guide calls this training-serving skew, and its advice is blunt: reuse the same feature code in both places, and log the exact features you served so you can train on them later.

A worked example. A model uses "orders in the last 7 days" per customer. During training the pipeline computes it from the full order history. At serving time the code reads a live table that has not loaded today's orders yet, so a customer who looked like three recent orders in training looks like one at serving. The model was never trained on that version of the input, and its score drifts off. This is why serious feature engineering, often backed by a feature store, becomes part of the serving story instead of an afterthought.

What to watch out for with model serving

Deploying the model is only the start. A model that scored well on day one decays as the world drifts away from its training data, so serving without monitoring is flying blind.

• Watch for data drift. Compare the distribution of live inputs against the training data. When they diverge you have data drift, predictions get less reliable, and the decay eventually shows up as model drift.

• Compare predictions to outcomes. Uptime dashboards prove the endpoint is answering. They say nothing about whether the answers are still correct. Once actual outcomes arrive, compare them to what the model predicted; Azure model monitoring, for example, tracks accuracy against ground truth and warns you before a stale model costs money.

• Keep versions traceable. You need to know which model version produced which prediction, which is where a model registry and experiment tracking earn their place.

• Make deployment repeatable. Serving belongs inside MLOps, with automated, versioned rollouts and a clean rollback path, not a manual copy of a file onto a server.