Model Monitoring in Production

Shipping a model is the start, not the finish. By the end of this lesson you'll detect data drift, track prediction quality over time, and wire up alerts that tell you to retrain — before users feel the damage.

Learn Model Monitoring in Production in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

A model that was 95% accurate on launch day can silently slide to 70% within weeks. Nothing in your code changed — the world changed , and your model didn't. There are two failures you must watch for.

Data drift — the inputs change shape. You trained on summer shoppers; now it's winter and the incoming feature values look different. The model still works correctly, it just hasn't seen this kind of input before.

Concept drift — the relationship between inputs and the right answer changes. A "good salary" meant 50k in 2010 and 80k today, so the same input should now produce a different label. The model is now answering the wrong question.

The trap: most teams only watch accuracy. But accuracy needs labels (the true answer), and labels often arrive weeks later — a problem called label lag . Input drift, by contrast, is visible the instant a request arrives. That's why you monitor both: drift warns you early, accuracy confirms the damage.

The simplest, label-free drift check is this: keep a reference batch (a sample of your training data) and, for each new current batch of live traffic, compare their summary statistics. If the mean (average) or standard deviation (how spread out the values are) move beyond a tolerance you set, the inputs have drifted.

Read the worked example below line by line — every function is plain Python, no libraries. Then run it and confirm the output matches the comment at the bottom.

Once labels do arrive, you can measure how well predictions are landing. But a single day's accuracy is noisy — one weird batch can make it jump or dip for reasons that don't matter. The fix is a rolling window : average the last few days so a genuine downward trend stands out from the daily wobble.

You then set an alert threshold — a line in the sand. When the rolling value crosses below it, you raise an alert (log it, post to Slack, or page on-call). The next example builds exactly that: a rolling accuracy that trips an alarm below 0.85.

The plain-Python examples show you the idea . In real systems you don't hand-roll the maths — you log every prediction and its inputs, push the numbers to a metric store, draw them on a dashboard, and let an alerting rule page you. A production stack usually layers up like this:

For the drift maths itself, purpose-built libraries do the heavy lifting:

Here's the same drift check from Section 2, but expressed with Evidently . It's read-only — the tool isn't installed in the editor — so study it as the production version of what you already built by hand.

Monitoring is only useful if it leads to action . The action is usually a retrain — fit a fresh model on recent data so it re-learns the world as it is now. Don't retrain on a single bad reading; tie it to a tiered, persistent signal:

The persistence rule ("sustained across a window") is what stops you retraining on a one-off spike — and it's the same idea as the rolling window you coded above.

Monitoring fails in predictable ways. Here are the four that bite teams most often:

The model ships and nobody watches it. Accuracy quietly rots and the first signal is an angry customer.

✅ Fix: log every prediction with its inputs from day one, even if the only "dashboard" is a daily print of mean/std and rolling accuracy.

You watch accuracy only. But accuracy needs labels, and by the time it drops the inputs have been drifting for weeks.

✅ Fix: monitor input statistics too — they're available immediately and warn you before accuracy moves.

You wait for ground-truth labels that take months to arrive (did the loan default? did the user churn?), so your alerts are always late.

✅ Fix: lean on label-free signals (input drift, prediction-distribution shift) as your early warning; treat accuracy as confirmation, not detection.

Every tiny wobble fires a page. The team mutes the channel — and then misses the alert that actually mattered.

✅ Fix: tier alerts (info / warning / critical), only page for critical, and require the signal to persist across a rolling window before firing.

Time to fade the scaffolding. You've detected drift and tracked rolling accuracy separately — now combine them into one daily watcher. The starter below gives you only a comment outline and the data. Write the loop yourself, then check it against the expected output in the comments.

Lesson complete — your models now have vitals and alarms!

You can tell data drift from concept drift, detect drift by comparing the mean and std of two batches, smooth prediction quality with a rolling-accuracy metric, trip an alert below a threshold, and decide when a sustained signal should trigger a retrain. That's the full monitoring loop, from raw signal to action.

🚀 Up next: MLOps Fundamentals — automate this whole loop so detection, retraining, and redeployment happen as a pipeline instead of by hand.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary