Linear Regression

Your first real machine-learning algorithm. By the end you'll fit a line to data by hand, measure its error with a cost function, watch gradient descent learn the line for you, split data to test honestly, and run it all in one line with scikit-learn.

Learn Linear Regression in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Imagine a seasoned estate agent who has watched hundreds of houses sell. Over time they develop a gut feeling: "bigger houses sell for more." Linear regression turns that gut feeling into an exact formula — something like price = 150 × size + 50000 .

The agent's instinct is fuzzy; the formula is precise and repeatable. That is the whole job of this lesson: take a scatter of dots and find the single straight line that best summarises the trend, so you can predict the next dot you haven't seen yet.

A linear-regression model is nothing more than two numbers. You feed it an input x (a feature , like study hours) and it returns a prediction y using the straight-line formula:

"Training" the model just means picking the w and b that draw the line closest to all your data points. Everything else in this lesson is about how we pick them. The example below starts with a hand-guessed line and a function that applies it.

How do you know one line is better than another? You need a single number that says "this is how wrong you are." That number is the cost function , and the standard one for regression is Mean Squared Error (MSE) :

For each point you take the gap between the real value and the model's guess (the error ), square it, then average across all points. Squaring forces every error to be positive (so a +5 and a −5 don't cancel) and makes big misses hurt much more than small ones. Lower MSE means a better fit. The worked example below prints each error and then the MSE.

Guessing w and b by hand doesn't scale. Gradient descent is the algorithm that finds them automatically. Picture the MSE as a valley: every (w, b) is a spot on the hillside, and the lowest point is the best line. Gradient descent is like walking downhill blindfolded — feel which way the ground slopes, take a small step that way, and repeat.

A model that scores its own homework will always claim an A. To get an honest measure of how it performs on data it has never seen, you split your data into two parts before training:

If the model does great on the training set but poorly on the test set, it has overfit — it memorised the specific points instead of learning the underlying trend. The test score is the number you actually trust. In real projects, train_test_split from scikit-learn does this split for you in one line.

You now understand what happens under the hood — so in real projects you let a library do it. scikit-learn (imported as sklearn ) gives you LinearRegression : create it, call .fit(X, y) to train, and .predict(...) to use it. It runs the maths far faster and more reliably than hand-rolled code.

Read the example below as a worked walkthrough — it needs scikit-learn installed locally to run, and the comments show the expected output. Notice the w and b it finds match what your gradient-descent loop learned above.

You report 99% accuracy, ship the model, and it fails on real users.

✅ Fix: hold back a test set the model never trains on, and judge it only by the test score. Use train_test_split(X, y, test_size=0.2) .

One feature is in the thousands (square feet) and another is single digits (bedrooms). Gradient descent zig-zags or diverges, and weights become hard to compare.

✅ Fix: scale features to a similar range first (e.g. StandardScaler ) before training. A smaller learning rate also helps stop the cost blowing up.

Your data curves, but you force a straight line through it. The MSE stays high no matter how long you train.

✅ Fix: plot the data first. If it bends, add polynomial features, transform a column (e.g. a log), or pick a model that can capture curves.

❌ Extrapolating far beyond the training range

You trained on houses 600–2000 sq ft, then predict a 10,000 sq ft mansion. The line keeps going forever, but reality doesn't.

✅ Fix: only trust predictions inside (or near) the range you trained on. Flag inputs far outside it as unreliable.

No blanks this time — just a brief and a comment outline. Write the two helper functions, score both candidate lines with MSE, and print which one fits better. Check your answer against the expected result in the comments.

You've built your first ML model from the ground up: you fit a line y = wx + b , scored it with the MSE cost function, wrote a gradient-descent loop that learns w and b on its own, understood why training and test sets matter, and saw how scikit-learn does it all in three lines.

🚀 Up next: Classification — instead of predicting a number on a sliding scale, you'll predict a category (spam vs not spam, cat vs dog), reusing this same predict-measure-improve loop.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary