Deep Learning Fundamentals

By the end of this lesson you'll be able to explain how a deep network turns numbers into predictions, how it learns from its mistakes with backpropagation and gradient descent, and how to build one in Keras without getting lost.

Learn Deep Learning Fundamentals in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Picture a factory assembly line. Raw materials enter at one end; each station does one small job and passes the result on. By the last station, scattered parts have become a finished car. A deep network works the same way — each layer is a station that refines the data a little before passing it along.

The first layers detect simple things (edges in an image, fragments of a word). Middle layers combine those into shapes or phrases. The final layer makes the call: "that's a golden retriever" or "this review is positive." Deep just means many of these stations stacked in a row — and more stations means the network can learn more abstract ideas.

A layer is a row of neurons. A network with one or two layers is called shallow ; a network with many layers is deep . The data flows forward: the output of one layer becomes the input to the next. This step — running data through every layer to get a prediction — is called a forward pass .

Two breakthroughs made deep networks practical: the ReLU activation (which keeps positive numbers and zeroes out negatives) fixed a problem where gradients vanished in deep stacks, and GPUs made it fast enough to train millions of weights. Below, you'll do a forward pass entirely by hand so the layers stop being a mystery.

Read every comment in this worked example, then run it. Each hidden neuron is just a weighted sum of the inputs, plus a bias, passed through an activation.

A fresh network guesses badly. Learning is the process of nudging every weight so the guesses get better. There are two parts:

The clearest way to feel gradient descent is on a single number. Below, you minimise f(x) = (x - 3)² . You already know the answer is x = 3 — watch the algorithm discover it by always stepping downhill, opposite the gradient.

Before a network can improve, it needs a single number that says how bad its current guesses are. That number is the loss . Gradient descent's whole job is to make this number smaller.

Different tasks use different loss functions:

These three dials control how training runs. They trip up beginners, so here they are in plain English:

You'd never hand-code backprop for a real model. Frameworks do the calculus and GPU work for you. The two most popular are TensorFlow/Keras (a model is a short list of layers — the gentlest start) and PyTorch (flexible and Pythonic, favoured in research). They teach the same ideas you just learned.

A big risk with deep models is overfitting : the network memorises the training data and then flops on new data. Regularisation fights this. The most common trick is dropout — during training it randomly switches off a fraction of neurons, forcing the network to spread its knowledge out instead of relying on a few memorised paths.

The Keras model below uses the same layers, ReLU, sigmoid, Adam optimiser, and binary cross-entropy loss from this lesson — plus one Dropout layer. Run it where TensorFlow is installed; here the expected summary is shown so you can read it as a reference.

These four problems trip up almost every beginner. Here's how to spot and fix them:

The loss jumps around, balloons to a giant number, or becomes nan . The steps are so big that gradient descent overshoots the minimum every time.

✅ Fix: lower the learning rate (try 0.001, then divide by 10 if it still diverges).

One feature ranges 0–1 and another ranges 0–100,000. Training is unstable or painfully slow because the large feature dominates every gradient.

✅ Fix: scale features to a similar range (e.g. subtract the mean and divide by the standard deviation) before training.

Training accuracy hits 99% but new-data accuracy is poor. The model memorised the training set instead of the pattern.

✅ Fix: add Dropout , get more data, or stop training earlier (early stopping).

With only a handful of examples, a deep network has nothing to generalise from and overfits instantly — it can't tell signal from noise.

✅ Fix: gather more data, use data augmentation, or pick a smaller/simpler model that needs fewer examples.

You've taken one step by hand and watched a worked loop. Now write the loop yourself from the outline below — no filled-in logic this time, just the plan.

Lesson 8 complete — you understand deep learning's engine!

You can run a forward pass by hand, explain backpropagation and gradient descent, choose the right loss function, set epochs/batches/learning rate sensibly, build a small Keras model, and fight overfitting with dropout. That's the core of every deep learning system in the world.

🚀 Up next: Natural Language Processing — teaching computers to understand and generate human text.

Practice quiz

What makes a neural network "deep"?

  • It has only one layer
  • It uses a large learning rate
  • It stacks many hidden layers
  • It has many output classes

Answer: It stacks many hidden layers. Depth means stacking many hidden layers, so early layers learn simple patterns and later ones combine them.

What does the ReLU activation do?

  • Keeps positive values and zeroes out negatives
  • Squashes values into 0..1
  • Returns the absolute value
  • Normalises to mean 0

Answer: Keeps positive values and zeroes out negatives. ReLU outputs the value if positive, else 0 — adding non-linearity while avoiding vanishing gradients in deep stacks.

What is a forward pass?

  • Sending the error backwards
  • Updating the weights
  • Computing the gradient
  • Running data through every layer to get a prediction

Answer: Running data through every layer to get a prediction. A forward pass runs the input through each layer in turn to produce the network's prediction.

What is backpropagation used for?

  • Loading the data
  • Computing each weight's gradient by sending error backwards
  • Initialising the weights
  • Choosing the batch size

Answer: Computing each weight's gradient by sending error backwards. Backprop uses the chain rule to send the error backwards, giving each weight a gradient of the loss.

In the gradient-descent update, which way does each weight move?

  • Opposite the gradient (downhill on the loss)
  • In the direction of the gradient
  • Randomly
  • Toward zero

Answer: Opposite the gradient (downhill on the loss). Weights step opposite the gradient — downhill — because the gradient points toward increasing loss.

Which loss function is standard for regression (predicting numbers)?

  • Cross-entropy
  • Hinge loss
  • Mean Squared Error (MSE)
  • Accuracy

Answer: Mean Squared Error (MSE). MSE squares the gap between prediction and target, so big mistakes are penalised heavily — ideal for regression.

What is one epoch?

  • One weight update
  • One full pass over all the training data
  • One layer of the network
  • One mini-batch

Answer: One full pass over all the training data. An epoch is one complete pass over the entire training dataset; a batch is a small chunk within it.

If the loss explodes to a huge number or NaN, the likely cause is:

  • Too many epochs
  • The batch size is too small
  • Dropout is too low
  • The learning rate is too high

Answer: The learning rate is too high. A too-high learning rate makes gradient descent overshoot, blowing the loss up; dividing it by 10 usually helps.

What is overfitting?

  • The model is too small to learn
  • The model memorises training data and does poorly on new data
  • The learning rate is zero
  • The data has no labels

Answer: The model memorises training data and does poorly on new data. Overfitting is memorising the training set instead of the general pattern, so it fails on unseen data.

How does dropout reduce overfitting?

  • It adds more layers
  • It increases the learning rate
  • It randomly switches off a fraction of neurons during training
  • It removes the bias terms

Answer: It randomly switches off a fraction of neurons during training. Dropout randomly disables some neurons each training step, forcing the network to spread out its learning.