Serving ML Models

Turn a trained model into a reliable production service — REST and gRPC inference endpoints, request batching, autoscaling, model versioning, and safe rollouts with canary and A/B deploys.

Learn Serving ML Models in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

A trained model in a notebook is like a chef who only cooks at home. Serving it is opening a restaurant: the same cooking skill, but now you must take orders, cook many at once, and never keep a table waiting.

The whole lesson is about running that kitchen well: fast tickets, full grills, enough cooks, and a safe way to change the menu.

Serving means putting your model behind a network address so other programs send an input and get a prediction back. The address is an inference endpoint .

A common setup is REST on the edge for callers, gRPC between internal services. Whichever you pick, two endpoints are non-negotiable: a /predict for inference and a /health the load balancer can poll.

Worked example — a minimal REST endpoint with FastAPI:

Notice the model loads once at startup and is reused by every request. The /health route is how the kitchen tells the maitre d' (load balancer) it is ready for orders.

FastAPI is great for one model, but a dedicated serving framework gives you batching, model versioning, multi-model hosting, and GPU scheduling for free. Pick by your stack:

A GPU runs a batch of 8 inputs almost as fast as a single one. Request batching groups incoming requests into one forward pass, so throughput (requests per second) shoots up. The cost: a request waits a few milliseconds for the batch to fill, so its latency rises slightly. Frameworks expose two dials — max batch size and max wait — to balance the two.

You measure latency in percentiles , not averages. p50 is the median request; p99 is the slow tail — 1 in 100 requests is at least this slow. Users feel the tail, so p99 is the number that matters for an SLA.

Worked example — the batching loop a framework runs for you:

The batch flushes when it is full or the wait timer fires — that timer is what keeps p99 from exploding under light traffic.

Run this to see how one slow request pulls p99 far above p50, even when most requests are fast. Read the comments, then press run.

Fill in the two blanks marked ___ . Group 7 requests into batches of 4 and count how many GPU passes that takes. Check your output against the # ✅ Expected output comment.

Fill in the two blanks so percentile() sorts the timings and returns the right value. A single 150ms request should send p99 sky-high while p50 stays calm.

Autoscaling adds or removes server replicas as traffic changes — more cooks at the dinner rush, fewer at 3am. You scale on a signal like GPU utilisation, queue depth, or requests per second.

The catch is the cold start : a freshly added replica is slow on its first request because the model still has to load into memory and the GPU warm up. Two fixes: keep a minimum number of replicas always running , and run a warmup inference before the replica accepts traffic (you saw the warmup call in Section 1).

Never overwrite a live model. Give every model a version (v1, v2, …) so you can serve a specific one, compare them, and roll back instantly if v2 misbehaves.

Trial the new recipe on a few tables (canary), or serve two recipes to see which sells better (A/B) — either way, the old recipe is one switch away.

These five mistakes sink most first serving deployments:

One request per forward pass leaves the GPU 90% idle and caps your throughput.

✅ Fix: enable dynamic batching (set max batch size + max wait), or use a framework that does it for you.

Loading the model inside the handler reloads 500MB+ from disk per call — 10s latencies.

✅ Fix: load once at startup, run a warmup inference, and keep a minimum replica count.

Overwriting the live model means a bad deploy has no undo and no way to compare.

✅ Fix: tag every model with a version and deploy via canary so rollback is one switch.

A synchronous DB or network call inside the handler stalls the whole worker under load.

✅ Fix: use async handlers, move slow work off the hot path, and set request timeouts.

An infinite request queue hides overload: latency climbs forever and memory blows up instead of failing fast.

✅ Fix: cap the queue length and reject extra requests with HTTP 429 so callers can back off.

Now write it yourself with only a comment outline. Build a tiny SLA checker that flags when your p99 latency breaches a 200ms budget. The starter has just the steps — no filled-in logic.

Lesson 42 complete — you can serve a model in production!

You can expose a model over REST or gRPC, pick a serving framework, batch requests for throughput, read p50/p99 latency, autoscale without cold starts, and roll out new versions safely with canary and A/B deploys.

🚀 Up next: Model Monitoring — watch your live model for drift, bias, and degradation before your users notice.

Practice quiz

What is model serving?

  • Training a model on more data
  • Compressing a model to fewer bits
  • Wrapping a trained model behind a network endpoint so other programs can get predictions
  • Cleaning the training data

Answer: Wrapping a trained model behind a network endpoint so other programs can get predictions. Serving exposes a trained model over a network (REST or gRPC) so clients can send inputs and receive predictions.

When is REST/JSON usually preferred over gRPC for inference?

  • When you want something easy to debug and call from any client, including browsers
  • When you need the lowest possible latency for tensors
  • When callers are only other internal services
  • When payloads are large binary blobs

Answer: When you want something easy to debug and call from any client, including browsers. REST/JSON is human-readable and easy to debug or call from a browser; gRPC suits fast internal service-to-service calls.

Why does request batching improve serving throughput?

  • It reduces model accuracy
  • It loads the model faster
  • It removes the need for a health check
  • A GPU runs a batch almost as fast as a single item, so grouping requests raises requests per second

Answer: A GPU runs a batch almost as fast as a single item, so grouping requests raises requests per second. Grouping many inputs into one forward pass uses the GPU efficiently, dramatically increasing throughput.

What is the main trade-off introduced by request batching?

  • Lower accuracy
  • A request may wait a few milliseconds for the batch to fill, raising its latency slightly
  • Higher memory usage forever
  • It disables versioning

Answer: A request may wait a few milliseconds for the batch to fill, raising its latency slightly. Batching raises throughput but a request waits for the batch to fill, slightly increasing per-request latency.

Why report latency as p50 and p99 percentiles instead of an average?

  • The slow tail (p99) is what users feel, and an average hides it
  • Averages are impossible to compute
  • Percentiles are always lower
  • p99 ignores slow requests

Answer: The slow tail (p99) is what users feel, and an average hides it. A few slow requests (the tail) matter most to users; p99 captures that tail while an average can hide it.

What is a cold start in model serving?

  • A request that returns an error
  • A request sent to the wrong version
  • The slow first request after a server boots, because the model must load and the GPU warm up
  • A batch that never fills

Answer: The slow first request after a server boots, because the model must load and the GPU warm up. A cold start is the slow first request after boot or scale-up while the model loads into memory.

Which practices help avoid cold starts?

  • Load the model inside every request handler
  • Load the model once at startup, run a warmup inference, and keep a minimum number of replicas
  • Scale to zero replicas always
  • Disable health checks

Answer: Load the model once at startup, run a warmup inference, and keep a minimum number of replicas. Loading once, warming up, and keeping minimum replicas running prevents the slow first-request penalty.

What does autoscaling do?

  • Versions the model automatically
  • Compresses the model
  • Batches requests
  • Adds or removes server replicas as traffic changes

Answer: Adds or removes server replicas as traffic changes. Autoscaling adjusts the number of replicas based on a signal like GPU utilisation or requests per second.

What is the difference between a canary deploy and an A/B deploy?

  • Canary measures which model is better; A/B is a safety check
  • Canary sends a small slice of traffic to a new version as a safety check; A/B splits traffic to measure which performs better
  • They are identical
  • Canary requires no rollback plan

Answer: Canary sends a small slice of traffic to a new version as a safety check; A/B splits traffic to measure which performs better. A canary is a safety mechanism (small slice, check health); an A/B deploy is a measurement mechanism comparing versions.

Why give every model a version when serving?

  • To make the model smaller
  • To increase latency
  • So you can serve a specific version, compare versions, and roll back instantly if a new one misbehaves
  • Because REST requires it

Answer: So you can serve a specific version, compare versions, and roll back instantly if a new one misbehaves. Versioning lets you compare models and instantly roll back a bad deploy instead of overwriting the live model.