Reinforcement Learning Basics

Master the core mechanics every RL algorithm is built on — the discounted return, value functions V(s) and Q(s,a), the Bellman equation, and the words used to describe how agents learn.

Learn Reinforcement Learning Basics in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Imagine someone offers you £100 today or £100 in a year . You take it today — money now is worth more than money later, because you could spend or invest it. RL agents feel exactly the same about reward.

The discount factor gamma (a number between 0 and 1) is the "interest rate" the agent applies to the future. A reward two steps away is multiplied by gamma², three steps away by gamma³, and so on. With gamma close to 1 the agent is patient and plans far ahead; with gamma close to 0 it is impulsive and grabs whatever reward is nearest. The return is just the total of all those discounted rewards — and maximising it is the agent's entire job.

A reward is the single number the environment hands you on one step. But a greedy agent that only chased the next reward would walk straight into traps. What you really care about is the return — the sum of all future rewards , with later ones discounted.

Read it left to right: take this step's reward in full, then add next step's reward shrunk by gamma, then the one after shrunk by gamma², and keep going. Because gamma is below 1, far-future rewards barely register — which also keeps the sum finite even if it never ends. The worked example below computes this for a real reward list and then does the one update step that turns an observed return into a learned value.

A single return is what happened in one run. A value function is the return you expect on average . There are two of them, and the only difference is how much they pin down:

"How good is it to be in state s?" The expected return if you start in s and act well from there.

"How good is it to do action a in state s?" The expected return if you take a now, then act well.

Q is the more useful of the two because it scores every action — so the agent can simply pick the action with the highest Q. And the two connect neatly: the value of a state is the value of its best action, written V(s) = max over actions Q(s, a) . The example computes both for a small two-action state.

The return G_t is an infinite sum, which sounds impossible to compute. The Bellman equation is the trick that makes it easy. It notices that the future, after one step, is just another value — so the whole sum folds into a tiny recursive rule:

In plain English: the value of where you are equals the reward you get now, plus the discounted value of wherever you land next (s′). You don't need to look at the entire future — you only need this step's reward and an estimate of the next state. That's the insight that turns RL from "sum over infinity" into a loop you can actually run.

RL problems come in two shapes, and they change how you think about the return:

Have a natural end — a chess game finishes, a robot reaches the goal, the player dies. The run is an episode ; afterwards everything resets . The return is a finite sum over the steps of that one episode.

Never stop — a thermostat, a stock-trading bot, a server tuning itself run forever . There's no reset and no final step.

This is the second reason discounting exists: for a continuing task the reward sum would run to infinity, but multiplying each future reward by gamma below 1 makes the total settle on a finite number. A terminal state (the end of an episode) is treated as having value 0 — there's no future left to discount.

A policy is the agent's strategy — its rule for choosing an action in each state. Algorithms differ in whether the policy they learn about is the same one they act with :

Fill in the two blanks so the program adds up the rewards with a discount factor of 0.5. Use the expected output to check yourself.

This is the learning rule at the heart of value-based RL. Fill in the blank to move the value estimate part of the way toward the observed return.

Discounting the first reward as well — every term ends up too small.

✅ Fix: step starts at 0, so the first reward uses gamma**0 = 1.

Setting gamma = 1 on a continuing (never-ending) task makes the return grow without bound.

✅ Fix: use gamma below 1 (0.9 or 0.99) for continuing tasks.

Writing (V_s - target) pushes the estimate away from the return — it diverges.

✅ Fix: the error is target minus current estimate.

Support is faded here — only the outline is given. Write the function and the averaging logic yourself, then check against the expected output in the comments.

Lesson complete — you now know the machinery of RL!

You can compute a discounted return, tell V(s) from Q(s,a), read the Bellman equation as plain English, separate episodic from continuing tasks, and explain on-policy vs off-policy. These are the exact pieces every RL algorithm assembles.

🚀 Up next: Q-Learning & DQN — turn the Bellman update into an algorithm that learns to play Atari from raw pixels.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary