Q-Learning & Deep Q-Networks

By the end of this lesson you'll be able to build a Q-table, apply the Bellman update by hand, and write an epsilon-greedy agent that learns the best path through a grid world — all in plain Python.

Learn Q-Learning & Deep Q-Networks in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Imagine you're dropped into a hedge maze with a chocolate bar hidden at the exit. You have no map. The first time, you wander randomly — left here, right there — until you stumble onto the exit and get the chocolate.

Each time you reach a junction (a state ) you remember which turn (an action ) eventually paid off. Over many runs you build a mental note for every junction: "from here, going right got me to the chocolate fastest." That mental note is exactly a Q-table — a score for every junction-and-turn pair.

Two instincts make you learn well. You discount faraway rewards (chocolate ten turns away is worth less than chocolate one turn away), and you sometimes explore a turn you've never tried, just in case there's a shortcut. Q-Learning is just this maze instinct written as a formula.

The heart of Q-Learning is the Q-table . Q(s,a) stores the agent's best guess of the total future reward it will earn if it takes action a in state s and then keeps acting optimally afterwards. "Q" stands for quality — the quality of that move.

You don't need a fancy data structure. A plain Python dictionary keyed by a (state, action) tuple works perfectly for small worlds. Every entry starts at 0.0 because the agent begins knowing nothing.

Every time the agent takes an action and sees what happens, it nudges one cell of the Q-table using this rule:

Read it piece by piece — in plain English it just says "move my old guess a little closer to a better guess":

It's called off-policy temporal-difference learning , but you don't need the jargon to use it. "Temporal difference" just means you learn from the gap between two estimates taken one step apart.

How big a step you take toward the new estimate.

How much you value future reward versus reward right now.

There's a tension at the core of learning. If the agent always picks the action it currently thinks is best (pure greedy), it can lock onto a mediocre path and never discover a better one. But if it acts randomly forever, it never cashes in what it learned.

Epsilon-greedy is the simple fix: with probability ε (epsilon) act randomly ( explore ), otherwise pick the best known action ( exploit ).

Here's the whole loop in miniature: a Q-table as a dictionary, a single Bellman update applied by hand, and an epsilon-greedy pick. Read every comment, run it, and watch Q(A,right) move from 0 to 5.

Fill in the two blanks so the update runs. The next state B already has learned values, so this time max_next won't be zero.

Complete the explore and exploit branches. Run it 1000 times and confirm the agent mostly picks the higher-value action.

Everything above scales to any small world. But a Q-table needs one cell per state-action pair, and that breaks down fast: an Atari screen has more possible pixel states than atoms in the observable universe. You can't store a table that big, and you'd never visit each cell enough to learn it.

A Deep Q-Network (DQN) swaps the table for a neural network. Instead of looking up Q(s,a) , the network predicts it from the raw state, so similar states share what they've learned — the GPS that generalises instead of the notebook that memorises. DeepMind's 2013 DQN learned to play Atari games straight from pixels using exactly this idea, plus two stabilisers:

The learning rule is still the Bellman update you mastered here — only the storage changed from a dict to a net.

These are the mistakes that quietly stop an agent from learning. None of them throw a Python error — the code runs, it just never gets good.

With alpha = 1.0 each update overwrites the old value with one noisy sample, so the Q-values jump around and never settle.

✅ Fix: use a small, steady α (0.1 is a safe default); lower it if values oscillate.

alpha = 0.001 barely moves the estimate each step, so training crawls and may not converge in the episodes you run.

✅ Fix: raise α (try 0.1) or run far more episodes. There's a trade-off — too high is unstable, too low is glacial.

gamma = 1.0 can stop the Q-values from converging at all; gamma = 0.1 makes the agent ignore the goal and chase tiny immediate rewards.

✅ Fix: stay in the usual 0.9–0.99 range so the agent plans ahead without diverging.

A pure-greedy agent commits to the first path it stumbles on and never discovers the better one — it gets stuck in a local optimum.

✅ Fix: keep epsilon > 0 so the agent keeps trying new actions while it learns.

Leaving epsilon = 0.5 for the whole run means half the agent's moves stay random even after it has learned the optimal path, capping its performance.

✅ Fix: start epsilon high and multiply it down toward a small floor (e.g. epsilon = max(0.05, epsilon * 0.99) ).

Trying to build a Q-table for chess or raw pixels means billions of cells — you run out of memory and never visit each state enough to learn it.

✅ Fix: when states can't be enumerated, switch from a table to a function approximator (a neural network) — that's a DQN.

Now put it all together with the support removed. The starter below gives you only a comment outline — no filled-in logic. Build the training loop yourself: a Q-table dict, epsilon-greedy moves, and the Bellman update running for 500 episodes until every state learns to head toward the goal.

Lesson complete — you can teach an agent to learn!

You built a Q-table from a dictionary, applied the Bellman update by hand, tuned α and γ, and wrote an epsilon-greedy agent that explores then exploits. You also saw exactly when a table runs out and a Deep Q-Network takes over.

🚀 Up next: Policy Gradient Methods — instead of learning action values, learn the policy directly. These are the algorithms behind PPO and ChatGPT's RLHF training.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary