Evaluating AI Models

By the end of this lesson you'll read a confusion matrix, pick the right metric for any task, and tell a great model from one that's only pretending.

Learn Evaluating AI Models in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Imagine grading a student with a single number — say "78% of answers correct." It hides a lot. Did they ace maths but fail science? Did the test even cover the right topics? A real report card breaks the grade into subjects, comments, and a comparison to the class, because one number can mislead.

Evaluating an AI model is exactly the same. One score — accuracy — can look brilliant while the model is useless. A proper evaluation is a report card: several metrics , each measuring a different subject, plus a check that you graded on questions the model never saw before. This lesson is how you write that report card.

When a model answers yes/no questions, every prediction lands in one of four buckets. A true positive is a correct "yes"; a true negative a correct "no". A false positive is a false alarm (said yes, was no), and a false negative is a miss (said no, was yes). Lay those four counts in a grid and you have the confusion matrix .

You don't need any library to build one — just walk the two lists together and tally. Run this:

Those four counts give you four very different views of quality. Accuracy is the fraction of all predictions that were right. Precision asks "when the model says yes, how often is it correct?" Recall asks "of everything it should have caught, how much did it?" And F1 blends precision and recall into one number using the harmonic mean, so a model can't cheat by being great at one and terrible at the other.

F1 = 2 · precision · recall / (precision + recall)

You hand-rolled the metrics above to understand them — but in real projects you call a library. Here is the exact same data scored by scikit-learn , including the ROC-AUC you can't easily do by hand. The # Expected output comment shows what you'd see if you ran it locally.

Notice the metrics match your hand calculations exactly. The library just saves you the arithmetic — and adds ROC-AUC, which needs the probability column y_prob .

Classification predicts a category ; regression predicts a number — a price, a temperature, an age. So instead of counting hits and misses, you measure how far off the predictions are.

MAE (mean absolute error) is the average size of the error in the data's own units. MSE squares the errors first, so big misses count extra; RMSE takes the square root of MSE to get back into the original units. R² reports the fraction of the data's variation the model explains — 1.0 is perfect, 0 is no better than always guessing the average.

Score a model on one random test split and you might just get lucky — or unlucky. k-fold cross-validation fixes that: it chops the data into k equal parts (folds), trains on k−1 of them and tests on the one held out, then rotates so every row gets tested exactly once . You report the average of the k scores.

The spread of those scores matters too. A small standard deviation means the result is stable; a big one means your model's quality swings with the split — a warning sign.

The whole reason you cross-validate is to catch two opposite failures. Underfitting (high bias ) is a model too simple to learn the pattern — it scores poorly on both the training data and new data. Overfitting (high variance ) is a model that memorised the training data, including its noise — it scores brilliantly on training but badly on anything new.

Time to fly solo. You're given a confusion matrix in the comments. Compute all four classification metrics from scratch — only the outline is provided, no formulas filled in.

These four traps catch almost every beginner. Spot them before they fool you.

If 99% of cases are negative, a model that always says "negative" hits 99% accuracy and catches nothing:

✅ Fix: report precision, recall, F1 and ROC-AUC alongside accuracy.

Scoring on the same data the model learned from rewards memorisation, not understanding:

✅ Fix: always score on a held-out test set the model has never seen.

❌ Reporting one lucky split, with no cross-validation

A single train/test split can hand you a flattering (or unfair) number by chance.

✅ Fix: use k-fold cross-validation and report the mean ± standard deviation, not one number.

Maximising precision on a cancer screen means you'll miss sick patients (low recall) to avoid false alarms — the opposite of what matters here.

✅ Fix: choose the metric from the cost of errors. Disease detection → recall. Spam filter → precision. Unsure → F1.

Lesson complete — you can now write a model's report card!

You can build a confusion matrix by hand, derive accuracy, precision, recall and F1, measure regression error with MAE/RMSE/R², cross-validate for a trustworthy score, and read the bias-variance gap to spot overfitting. Most importantly, you know that the metric you choose shapes the model you build.

🚀 Up next: Model Compression — make a good model smaller and faster without losing the quality you just learned to measure.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary