Policy Gradient Methods

By the end of this lesson you'll understand how an agent can learn what to do directly — turning action preferences into probabilities, sampling actions, and nudging the policy toward higher reward.

Learn Policy Gradient Methods in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Imagine two tennis players. The first, a value-based player, mentally scores every possible shot before each swing — "a cross-court forehand is worth 7, a drop shot is worth 4" — then picks the highest score. It's accurate but slow, and it falls apart when there are infinitely many shot angles to rate.

The second, a policy-based player, just builds instincts : "when the ball comes here, I usually go down the line." They don't rate options — they directly adjust how often they play each shot based on whether it tends to win the point. That direct adjustment of action probabilities is exactly what a policy gradient does.

A policy (written π, "pi") is the agent's rule for choosing actions: given a state, it says how likely each action is. Written formally it's π(a|s) — "the probability of action a in state s ."

Value-based methods like Q-learning never store a policy directly. They learn a value for each action, then act greedily on those values. Policy-based methods flip that around: they store the policy itself and adjust it to earn more reward, never bothering to rate actions on an absolute scale.

The simplest policy stores one number per action — a preference . To turn raw preferences into a valid probability distribution (all positive, summing to 1), you run them through the softmax function. Bigger preference means bigger probability, but every action keeps a non-zero chance, so the agent still explores.

Run this worked example and watch the preferences become probabilities:

Once you have probabilities, the agent doesn't always pick the top one — it samples . Python's random.choices does weighted sampling: pass the probabilities as weights and it returns higher-probability actions more often, but not every time. That built-in randomness is how policy gradient agents explore.

This example samples 1,000 actions and shows the frequencies matching the target probabilities:

REINFORCE is the simplest policy gradient algorithm, and the rule is intuitive: take an action, see the reward, then make that action more likely if the reward was good and less likely if it was bad. Repeat millions of times and the policy drifts toward whatever earns reward.

The objective being maximised is the expected reward , written J(θ) = E[ R ], where θ ("theta") are the policy's parameters (your preferences). The famous policy gradient theorem says you can climb this objective with the update:

In plain English: nudge each parameter in the direction that increases the log-probability of the action you took, scaled by the reward R . Positive reward pushes the action up; negative reward pushes it down. The worked code below does exactly this for one action, and you'll do an update by hand in the next exercise.

Plain REINFORCE works, but it's noisy . Because it uses the raw reward of a whole episode, the learning signal swings wildly — two nearly identical episodes can give very different totals. That high variance makes training slow and unstable.

The fix is the advantage : instead of using the raw reward, subtract a baseline (a reference reward, often the average) and use the difference:

Now an action is judged as better or worse than expected , not on its raw, noisy total. This slashes variance without biasing the gradient — actions that beat the baseline go up, those below it go down.

Actor-critic methods take this one step further. They run two pieces side by side: the actor is the policy that chooses actions, and the critic is a learned value function that estimates how good each state is. The critic's estimate becomes the baseline for the actor's update. This is the foundation of A2C and PPO — the algorithm used to fine-tune ChatGPT with RLHF — which is why those methods learn far more stably than plain REINFORCE.

These are the four mistakes that derail almost every first policy gradient implementation:

❌ Training is unstable and never converges (high variance)

Plain REINFORCE uses the full episode return, which swings wildly from run to run. The gradient estimate is so noisy the policy can't settle.

If every reward in your task is positive (say 0 to 100), every action gets pushed up — the policy can't tell good actions from merely-less-good ones.

❌ Learning rate too high — the policy collapses

A large learning rate lets one lucky episode shove all the probability onto a single action. The policy stops exploring and gets stuck forever.

Rewards of +1000 produce giant gradients; rewards of 0.0001 produce no learning. Either way the update size is wrong.

Time to put it all together with the support faded. The starter below is just a comment outline — write the training loop yourself. Use softmax for the policy, random.choices to sample, and an advantage (reward minus a running baseline) for the update.

Lesson complete — you can think in policies now!

You learned how policy-based RL differs from value-based RL, how softmax turns preferences into probabilities, how to sample an action, the REINFORCE update rule and its objective, and why advantages, baselines, and actor-critic methods make training stable. These ideas scale all the way up to PPO and the RLHF that aligns modern language models.

🚀 Up next: Computer Vision Pipelines — build end-to-end vision systems for real-world applications.

Practice quiz

What does a policy-based RL method learn directly?

  • A value for each state only
  • The reward function
  • The policy itself — the probability of each action in each state
  • The transition probabilities

Answer: The policy itself — the probability of each action in each state. Policy-based methods adjust action probabilities directly, rather than learning action values first.

How does value-based RL (like Q-learning) differ from policy-based RL?

  • Value-based learns action values then acts greedily; policy-based learns the policy directly
  • Value-based has no rewards
  • Policy-based cannot explore
  • They are the same approach

Answer: Value-based learns action values then acts greedily; policy-based learns the policy directly. Q-learning learns Q(s,a) and acts greedily; policy gradients tune action probabilities directly.

What does the softmax function do to action preferences?

  • Picks only the largest preference
  • Sets every probability to zero
  • Sorts them in order
  • Turns them into a probability distribution that is all positive and sums to 1

Answer: Turns them into a probability distribution that is all positive and sums to 1. Softmax exponentiates and normalizes preferences so they form a valid probability distribution.

Why does a softmax policy keep exploring?

  • It always picks a random action
  • Every action keeps a non-zero probability, so the agent still tries others
  • It ignores preferences
  • It never updates

Answer: Every action keeps a non-zero probability, so the agent still tries others. Even the best action does not get probability 1, so lower-preference actions still get chosen sometimes.

What is the core idea of the REINFORCE algorithm?

  • Make actions that earned high reward more likely and low-reward actions less likely
  • Always pick the action with the highest Q-value
  • Never change the policy
  • Memorize the entire reward table

Answer: Make actions that earned high reward more likely and low-reward actions less likely. REINFORCE nudges the policy so high-reward actions become more probable and low-reward ones less probable.

The policy gradient update is proportional to which quantity?

  • The number of states
  • The learning rate alone
  • Reward times the gradient of the log-probability of the chosen action
  • The discount factor squared

Answer: Reward times the gradient of the log-probability of the chosen action. The policy gradient theorem gives an update of R times the gradient of log pi(a|s).

Why does plain REINFORCE have high variance?

  • It uses too small a learning rate
  • It uses the raw return of a whole episode, which swings wildly between runs
  • It never receives rewards
  • It only uses one state

Answer: It uses the raw return of a whole episode, which swings wildly between runs. Episode returns are noisy, so the gradient estimate is noisy and learning is slow and unstable.

What is the advantage in policy gradient methods?

  • The raw reward times two
  • The number of episodes run
  • The maximum Q-value
  • Reward minus a baseline, judging an action as better or worse than expected

Answer: Reward minus a baseline, judging an action as better or worse than expected. advantage = reward - baseline; subtracting a baseline cuts variance without biasing the gradient.

Why does subtracting a baseline help training?

  • It makes all rewards positive
  • It dramatically lowers variance without biasing the gradient
  • It removes the need for a policy
  • It speeds up the environment

Answer: It dramatically lowers variance without biasing the gradient. A baseline reframes actions as better-or-worse-than-expected, reducing noise in the gradient estimate.

In actor-critic methods, what role does the critic play?

  • It chooses the actions
  • It deletes the policy
  • It estimates state values, serving as the baseline for the actor's policy update
  • It stores the replay buffer

Answer: It estimates state values, serving as the baseline for the actor's policy update. The actor is the policy; the critic's value estimate is the baseline that cuts variance (used in A2C and PPO).