Diffusion Models Explained

By the end of this lesson you'll be able to explain — and run code for — how DALL-E and Stable Diffusion turn random noise into images by learning to denoise, step by step.

Learn Diffusion Models Explained in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Picture an old TV tuned to a dead channel — a screen full of fuzzy static . Now imagine a sculptor who can look at any patch of static and carefully chip away the randomness until a clear picture appears underneath. That is exactly what a diffusion model does: it sculpts an image out of noise .

During training the model watches the opposite happen thousands of times — real photos slowly dissolving into static — so it learns precisely what "noise" looks like at every stage. To generate something new you hand it a fresh screen of static and your text description, and it removes the noise a little at a time, guided by your words, until a finished image emerges.

Diffusion models power DALL-E, Stable Diffusion, Midjourney, and Imagen. They produce higher-quality images than older GANs with far more stable training, and the same denoising recipe also generates audio, video, and 3D shapes.

The forward process takes clean data and adds a small amount of Gaussian noise (bell-curve random values) at each of many timesteps. Repeat it enough and the data becomes indistinguishable from pure noise. Crucially this process is fixed , not learned — it follows a preset noise schedule (called beta ) that decides how much noise to add at each step.

Why deliberately destroy data? Because it creates perfect training pairs: at every step you know exactly how much noise you added, so you can teach a network to undo it. Run the worked example below to watch a tiny 5-value signal dissolve into noise.

The reverse process is the part the model actually learns. Given a noisy sample, the network predicts the noise that was added, and you subtract that prediction to get a cleaner sample. The network is usually a U-Net — an encoder-decoder shape with skip connections that's great at processing images — but for the idea you only need one move: predict the noise, then subtract it.

The example below fakes a decent noise prediction so you can see a single denoise step pull a noisy signal back toward the clean one.

Sampling is how you create something new. You start from a screen of pure random noise and apply the denoise step over and over, walking backwards through the timesteps until a clean sample appears. More steps generally means higher quality but slower generation — early models used 1,000 steps, while DDIM and modern samplers get great results in 20-50.

The toy loop below starts from random noise and converges toward a target shape. A real model never sees the target; it learned that shape from millions of training images.

Training is surprisingly simple. Take a clean sample, pick a random timestep, add the matching amount of Gaussian noise, and ask the network to predict that noise . The loss is the mean squared error between the predicted noise and the true noise. Because you generated the noise yourself, you always have a perfect answer to grade against — no human labels needed. This is self-supervised learning.

Run the example to compare an untrained model (predicts nothing) against a trained one (predicts the noise accurately) and see the loss drop.

So far the model denoises toward any realistic image. To get the image you want, you condition the denoising on a text prompt. The prompt is encoded (often by a model like CLIP) and fed into the U-Net through cross-attention , so every denoise step is nudged toward "a cosy cabin in a snowy forest" rather than just anything.

Classifier-free guidance (CFG) makes the prompt count more. At each step the model predicts noise twice — once with the prompt and once with no prompt — then exaggerates the difference between them. The guidance scale controls how much: around 7-8 is the usual sweet spot.

Denoising every pixel of a 512×512 image, hundreds of times, is slow. Latent diffusion fixes this by doing the whole process in a small latent space . An autoencoder first compresses the image down to a tiny representation (e.g. 64×64), diffusion runs there, and then a decoder expands the cleaned latent back to a full image.

This is the trick behind Stable Diffusion : it denoises something roughly 48× smaller, so it runs on a consumer GPU instead of a data centre. The text conditioning and guidance you just learned happen inside that latent space.

The worked example below shows the real Hugging Face diffusers API. It needs diffusers , torch , and a GPU, so treat it as a preview of the real code — the expected output is shown in a comment.

The predict-the-noise recipe works on anything you can add noise to. Swap the data and the network shape, keep the forward/reverse idea, and you get a generator for a new domain:

❌ Output is blurry or noisy — too few sampling steps

You set num_inference_steps very low, so denoising didn't finish.

❌ Image ignores the prompt OR looks "fried" — wrong guidance scale

Guidance too low ignores your words; too high oversaturates and distorts.

❌ Garbage samples — train/sample noise-schedule mismatch

You sampled with a different number of steps or schedule than you trained on.

Pixel-space or large batches won't fit; latent diffusion plus half precision helps.

Time to fade the scaffolding. Add noise to a clean signal over a few steps, then take one denoise step using a noise prediction you compute yourself, and show that you land back near the original. The starter gives you only a comment outline — write the logic yourself.

Lesson complete — you can now explain how diffusion models work!

You learned that the forward process adds Gaussian noise to data, that the network is trained to predict that noise with an MSE loss, and that sampling starts from pure noise and denoises step by step. You also saw how text conditioning plus classifier-free guidance steer the output, and how latent diffusion makes Stable Diffusion fast enough for a consumer GPU.

🚀 Up next: LLM Architecture — see how the same "predict the next piece" idea powers large language models.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary