Semantic Segmentation

Label every pixel in an image. By the end you'll understand encoder-decoder networks like U-Net, compute IoU and Dice by hand, and know when each segmentation flavour and metric is the right tool.

Learn Semantic Segmentation in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Object detection draws rectangles around things. Segmentation goes further: it hands you a colouring book of the photo and asks you to colour every single pixel by what it is — sky blue, road grey, car red, person yellow. Nothing is left uncoloured, and the colour has to follow the object's exact outline, not a loose box.

A self-driving car needs that precision to know exactly where the road ends and the kerb begins. A radiologist needs it to trace the precise boundary of a tumour. That is the job of semantic segmentation : assign a class label to each pixel so the whole image becomes a clean, colour-coded map.

In image classification you predict one label for the whole image ("this is a cat"). In segmentation you predict one label for every pixel . The output is a grid the same height and width as the input, where each cell holds a class id — a mask .

Under the hood the model produces, for each pixel, a score per class. You then take the argmax (the index of the highest score) to get that pixel's predicted class. That single operation is what turns a stack of score maps into the final colour-coded picture.

Here is that whole idea in plain Python — no libraries — so you can see the argmax happen on a tiny grid:

There are three flavours of segmentation. They answer different questions:

Every pixel gets a class, but all objects of a class merge. Three cars are just one big "car" region. Question: what is this pixel?

Separates individual objects (car #1, car #2, car #3) with their own masks, but usually ignores background "stuff". Question: which object is this?

Both at once: every pixel gets a class and countable objects get a unique id. Question: what is it, and which one?

Jargon check: classes like road, sky and grass that you can't count are called "stuff" ; countable classes like cars and people are "things" . Semantic handles stuff well; instance handles things; panoptic does both.

A normal classifier ends in dense layers that throw away spatial layout — fine for one label, useless for a per-pixel map. The Fully Convolutional Network (FCN) fixed this by replacing those dense layers with convolutions, so the output stays a 2-D grid.

Modern segmentation networks share an encoder-decoder shape:

U-Net (2015) is the famous example. Its big idea is skip connections : at each decoder stage it concatenates the matching high-resolution feature map copied straight from the encoder. That restores the fine edge detail the encoder threw away, so boundaries come out crisp instead of blurry. The diagram below shows the symmetric "U" shape:

The worked example below builds a tiny U-Net in PyTorch so you can see the skip connection ( torch.cat ) and the upsampling step in real code. It's read-only here — run it locally with PyTorch installed:

Notice nn.ConvTranspose2d doing the learnable upsampling and torch.cat([x, s1], dim=1) performing the skip connection. The 1×1 head maps features to one score per class per pixel.

The bottleneck feature map is tiny — maybe 16×16 for a 256×256 input. The decoder must grow it back to full size. There are two common ways:

Either way, the goal is the same: recover the original height and width so you can output one prediction per input pixel. DeepLab takes a different route and uses dilated (atrous) convolutions — convolutions with gaps between weights — to widen the receptive field without downsampling as aggressively, keeping resolution high throughout.

How do you score a mask? You compare the predicted pixels for a class against the true pixels. Two measures dominate:

Mean IoU (mIoU) is just the IoU averaged across all classes — the headline number on datasets like Cityscapes and Pascal VOC. The runnable example below computes both IoU and Dice on two tiny masks, entirely in plain Python so you can follow every pixel:

You almost never hand-build U-Net for real work. Libraries like segmentation_models_pytorch give you battle-tested architectures with pretrained encoders in a few lines. Read-only here; run it locally:

Swap encoder_name for a deeper backbone, or smp.Unet for smp.DeepLabV3Plus , and the rest of your training loop stays identical.

Fill in the two blanks so the loop counts how many pixels were labelled correctly, then divides by the total. The expected output is in the comments — match it.

Complete the argmax comparison and the list comprehension so each pixel's highest-scoring class becomes its label. Match the expected output.

Time to fly with less support. Using only the comment outline, compute the IoU for each of the three classes and then their mean. The expected numbers are in the comments so you can self-check.

Background often fills 80–95% of an image. Train with plain cross-entropy and the model learns to predict "background" everywhere — small classes vanish.

✅ Fix: use Dice loss or a weighted/focal cross-entropy so rare classes count more, and always report per-class IoU, not just the average.

A model scoring 92% pixel accuracy can still miss every pedestrian if pedestrians are a tiny fraction of pixels.

✅ Fix: judge models by mean IoU (mIoU), which weights every class equally regardless of how many pixels it covers.

Edges of objects come out smeared and the mask leaks across object borders — usually because spatial detail was lost during heavy downsampling.

✅ Fix: use an architecture with skip connections (U-Net) or dilated convolutions (DeepLab) so high-resolution detail reaches the output.

A pure encoder-decoder with no skips upsamples only from the tiny bottleneck, so it never recovers fine structure and masks look like soft blobs.

✅ Fix: concatenate (or add) the matching encoder feature map into each decoder stage — that is exactly what makes U-Net work.

Lesson 33 complete — you can label every pixel!

You now know the three segmentation flavours, how encoder-decoder networks like FCN and U-Net rebuild a full-resolution mask, why skip connections and transposed convolutions matter, and how to compute IoU, Dice and mIoU by hand.

🚀 Up next: Speech Recognition — switch from pixels to audio and learn how models turn sound waves into text.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary