Residual Networks & DenseNets

Learn the one idea — the skip connection — that let neural networks go from a few dozen layers to hundreds, and how to put a pretrained ResNet to work on your own images in minutes.

Learn Residual Networks & DenseNets in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Picture a 100-stop bus route across a city. A regular passenger has to get off and back on at every single stop — by the time they reach the end, the journey is exhausting and a lot has been forgotten. That is a deep plain network : information is reprocessed at every layer, and the original signal slowly gets garbled.

Now imagine an express lane running alongside the route — a shortcut that carries the original passenger straight through, untouched, while the local bus still makes its stops. At each stop you simply add whatever the local route figured out to the passenger who took the express lane. That shortcut is a skip connection . The express lane keeps the original information intact, so the network can be enormously deep without losing the thread.

ResNet builds one express lane per block (it adds the shortcut). DenseNet is even more generous: every stop gets its own dedicated lane to every later stop, and the lanes are bundled together (it concatenates ) so nothing is ever thrown away.

You'd expect a deeper network to be at least as good as a shallow one — worst case, the extra layers could just copy their input and do no harm. In practice, before 2015, the opposite happened. Stack enough plain layers and accuracy got worse , on the training set as well as the test set. A famous result showed a 56-layer network beaten by a 20-layer one.

This is the degradation problem , and it is not overfitting (overfitting would mean great training accuracy, poor test accuracy). It's an optimisation problem: as a gradient travels backwards through dozens of layers during training, it gets multiplied at every step. Multiply by small numbers enough times and it shrinks to almost nothing — the vanishing-gradient problem — so the early layers receive almost no signal telling them how to improve.

ResNet's idea is almost embarrassingly simple. Instead of asking a block to produce the full desired output H(x) , ask it to produce only the residual — the change — and then add the original input back:

The + x term is the skip connection (also called a shortcut or identity connection). It costs no extra parameters — it's just addition. Why does it help so much? Because if the best thing a block can do is leave its input alone (an identity mapping ), it just needs to drive F(x) towards zero. Pushing weights towards zero is easy; learning to perfectly reproduce the input with raw weight layers is hard. The skip guarantees deeper networks can never be worse than shallower ones — the extra blocks can always fall back to doing nothing.

The example below builds one residual block in plain Python so you can see F(x) + x directly:

Here's the heart of why ResNet works. When the training signal flows backwards, a plain layer multiplies the gradient by some factor. Across many layers those factors compound — and if they're below 1, the gradient vanishes. A skip connection changes the maths: because output = F(x) + x , the gradient gets a guaranteed "+1" path at every block, so it can never fully die.

The runnable example below contrasts the two. The plain column shrinks towards zero as depth grows; the residual column stays healthy:

A 3x3 convolution on many channels is expensive. To build really deep ResNets (50, 101, 152 layers) without the cost exploding, ResNet uses a bottleneck block : a three-step sandwich.

The narrow middle is the "bottleneck". By doing the heavy 3x3 work on far fewer channels, the block delivers similar power for a fraction of the computation — which is exactly how ResNet-50 stays trainable on ordinary hardware. The skip connection still wraps the whole sandwich: output = bottleneck(x) + x .

The skip's x and the block's output must be the same shape to add them. When a block changes the channel count or downsamples (stride 2), ResNet puts a 1x1 "projection" convolution on the skip path so both sides line up. Forgetting this is the #1 build error (see Common Errors).

DenseNet (2017) pushed the skip idea to its limit. Where ResNet adds the input ( F(x) + x ), DenseNet concatenates : inside a dense block, every layer receives the stacked feature maps of all earlier layers as its input. Layer 4 sees the outputs of layers 1, 2, and 3 bundled together.

Each layer contributes only a small fixed number of new feature maps — the growth rate k (often 12 to 32). Because every layer can reach back to all earlier features, the layers themselves can be narrow, which keeps the total parameter count surprisingly low. Concatenation also gives every layer a direct line to the loss, so gradients flow even more freely than in ResNet.

Because channels keep growing, DenseNet inserts transition layers (a 1x1 conv plus pooling) between dense blocks to compress the feature maps back down and keep memory in check.

You almost never train a ResNet from scratch. A ResNet trained on ImageNet (1.2 million images, 1000 classes) has already learned reusable visual features — edges, textures, shapes, object parts. Transfer learning reuses those learned features: load the pretrained weights, swap out the final classification layer for your own classes, and fine-tune. The two read-only examples below run locally with torchvision installed ( pip install torch torchvision ).

Load a pretrained model and classify an image:

Adapt it to your own classes by replacing only the final layer:

❌ Degradation: stacking plain layers without skips

Building a very deep network out of plain conv layers and finding it trains worse than a shallow one. The gradient vanishes before it reaches the early layers.

You change channels or downsample inside the block, then try to add the original input — and PyTorch raises a shape error because the two tensors don't match.

❌ Training from scratch instead of using pretrained weights

Calling resnet50() with no weights gives a randomly initialised network. On a small dataset it trains slowly and underperforms because it has to relearn basic vision from nothing.

Fine-tuning all 25M parameters on a few hundred images: training accuracy hits 100% while validation accuracy stalls. The model memorised the data instead of learning it.

Time to fly solo. Build a residual block from scratch and pass a signal through it ten times. The starter block gives you the brief only — write the logic yourself, then check it against the expected output.

Lesson complete — you understand what made deep networks possible!

You can explain the degradation problem, build a residual block ( output = F(x) + x ), describe why the identity shortcut and bottleneck make 100+ layer networks trainable, contrast ResNet's addition with DenseNet's concatenation, and fine-tune a pretrained ResNet with transfer learning.

🚀 Up next: Training Stability Techniques — the methods (batch norm, learning-rate schedules, and more) that keep these deep networks training smoothly.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary