Decision Trees & Random Forests

Learn exactly how a tree chooses its questions, why deep trees overfit, and how random forests combine many trees into one of the most reliable models in machine learning.

Learn Decision Trees & Random Forests in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

A decision tree works just like the game 20 Questions . To guess an animal you ask: "Is it bigger than a cat?" → Yes → "Does it live in water?" → No → "Does it have stripes?" → Yes → "It's a tiger!" Each question splits the remaining possibilities into smaller groups until only one answer is left.

A good player asks the most useful question first — the one that rules out the most options. A decision tree does the same thing automatically: at every step it picks the question that separates the classes best. The rest of this lesson is really just "how does the tree decide which question is best?"

A decision tree is built from nodes . Each internal node asks one question about a single feature (like "is temperature 23?"). The answer sends each row down a branch . When a group is "good enough" — mostly one class — the tree stops and that group becomes a leaf , which holds the prediction.

To choose a question, the tree needs to measure how mixed a group is. A group of all "yes" rows is pure and needs no further questions; a 50/50 mix of "yes" and "no" is the most impure . The two standard ways to measure this are Gini impurity and entropy .

Gini impurity is the chance you'd label a random row wrong if you guessed based only on the group's class mix. The formula is 1 − Σ pᵢ² , where pᵢ is the fraction of each class. It is 0.0 for a pure group and 0.5 for a 50/50 two-class mix . To score a split , you take the impurity of each side and weight it by how many rows land there — the side with more rows counts for more.

Run this worked example and watch the numbers. Lower weighted Gini means a better split.

Entropy measures the same idea as Gini — how mixed a group is — but with logarithms: −Σ pᵢ·log₂(pᵢ) . It is 0 for a pure group and 1.0 for a perfect 50/50 two-class split. Information gain is simply how much the entropy drops after a split: the entropy before, minus the weighted entropy of the groups after. The tree picks the feature with the highest information gain .

Gini and entropy almost always agree on which split is best. Gini is a touch faster (no logs), which is why it's scikit-learn's default; entropy/information gain is the classic textbook framing.

In practice you don't hand-code splits — you call a library. Here's the same idea with scikit-learn's DecisionTreeClassifier . Study it (it isn't runnable in the in-browser sandbox), then notice how the concepts map: criterion="gini" is the impurity measure you just computed, max_depth=2 limits how deep the tree can grow (your anti-overfit dial), and feature_importances_ tells you which inputs mattered.

The expected output is shown in the comments. Notice temp has importance 1.0 and humidity 0.0 — this tiny tree solved the problem using temperature alone, so humidity contributed nothing.

Left unchecked, a tree keeps splitting until every leaf holds a single training row. That tree scores nearly 100% on the data it learned from — but it has memorised the training set, including its random noise, instead of learning the real pattern. This is overfitting , and it shows up as a big gap between training accuracy and test accuracy.

You keep a tree general by limiting its complexity — this is pruning :

Because a tree explicitly chooses which feature to split on, it can tell you which inputs mattered most . Each feature's importance is the total impurity it removed across all the splits that used it, scaled so the scores add up to 1.0. A feature that's never used scores 0.0.

This is one of the biggest reasons people love trees: they're interpretable . A doctor, a loan officer, or a regulator can read the importances and the split thresholds and understand exactly why the model said what it said — something a neural network struggles to offer.

One deep tree overfits easily. A random forest fixes that by training many trees and letting them vote. Two tricks make the trees different from each other:

Each tree makes different mistakes. When you take the majority vote (for classification) or the average (for regression), the random errors cancel out and the shared signal remains. The forest is almost always more accurate and far more stable than any single tree — which is why random forests are a go-to baseline for tabular data.

Fill in the two blanks so gini() returns the impurity of a group. Remember: each p is a class's fraction, and you subtract p² from 1. The expected output is in the comments so you can self-check.

A decision tree is ultimately a stack of if statements. Here you write the prediction rule of a tree with a single decision node. Fill in the two blanks so the rule predicts correctly for hot and cool days.

Now do it with the support faded. You're given two candidate splits of the same six rows. Write the code to compute each split's weighted Gini and print which one the tree should pick (lower wins). Only a comment outline is provided — the logic is up to you.

These are the traps that bite almost everyone working with trees. Watch for them.

A fully grown tree memorises the training set: 100% train accuracy, poor test accuracy.

✅ Fix: cap the depth (and/or set a minimum leaf size):

If you only look at training accuracy you'll never notice overfitting, so you'll never prune.

✅ Fix: always compare against held-out test data:

Standardising inputs is required for k-NN/SVM, but for trees it's pointless effort and can confuse your pipeline.

✅ Fix: feed trees the raw values — no scaling needed:

❌ Data leakage — letting the answer sneak into the features

If a feature secretly encodes the label (or you fit on the test set), accuracy looks amazing in training and collapses in production.

✅ Fix: drop anything you wouldn't know at prediction time, and split before fitting:

You now understand how a tree turns data into questions: it measures impurity with Gini or entropy , picks splits by information gain , controls overfitting with depth limits and pruning , reports feature importance , and scales up to a random forest that votes many trees into one robust answer.

🚀 Up next: Neural Networks — where the model learns its own features instead of splitting on the ones you give it.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary