Policy Gradient Methods Explained (Reinforcement Learning Basics)

🤖 Policy Gradient & Function Approximation in Reinforcement Learning

Reinforcement Learning (RL) is transforming industries—from robotics to gaming and beyond. At the heart of modern RL lies a powerful combination: policy gradient methods and function approximation. This guide explains what they are and how they work together to solve real-world problems.

🧠 Policy Gradient Methods: A Quick Refresher

A policy defines how an agent behaves. It maps observed states (e.g., position, speed) to actions (e.g., move left or right).

1. Sample actions from the current policy
2. Observe rewards from the environment
3. Update the policy parameters to increase rewards

Instead of evaluating all actions, policy gradient methods directly increase the probability of good actions.

🔗 Beginner guide: A Beginner’s Guide to Policy Gradient

🧩 Function Approximation: Why It’s Crucial

In complex environments with continuous variables (angles, velocities, forces), storing every state–action pair in a table is impossible.

• Generalization – learn once, apply everywhere
• Scalability – handle huge state spaces
• Continuous control – real-world friendly

🔗 Deep dive: Function Approximation in RL

🔗 How They Work Together

The policy is represented by a neural network:

• Input: environment state
• Output: action probabilities

The network parameters define the agent’s behavior.

gradient = average(reward × ∇ log(policy))

Actions that produce higher rewards are reinforced.

Learning transfers to unseen states—flat ground → uneven terrain, simulation → real world.

💻 CLI Training Example

$ python train_policy.py Episode: 120 Average Reward: 245.7 Policy Loss: -0.032 Value Loss: 0.41 Policy updated successfully ✔

🌍 Real-World Applications

• PPO – stable and efficient continuous control
• DDPG – precision tasks like robotic arms
• SAC – balances exploration and exploitation

These power systems like AlphaGo and robotic manipulation.

💡 Key Takeaways

• Policy gradients directly optimize decision-making
• Function approximation enables real-world scale
• Neural networks make continuous control possible
• This combo powers modern deep reinforcement learning

Reinforce vs Actor-Critic: Key Reinforcement Learning Algorithms Explained

Understanding Reinforce vs Actor-Critic in Reinforcement Learning

Reinforcement Learning (RL) is one of the most exciting areas of Machine Learning because it focuses on how intelligent agents learn through interaction. Unlike supervised learning, where models learn from labeled datasets, Reinforcement Learning allows an agent to learn by trial and error. The agent explores an environment, takes actions, receives rewards or penalties, and gradually improves its strategy.

Among the many algorithms in Reinforcement Learning, two important approaches are Reinforce and Actor-Critic. Both belong to the family of policy optimization methods, but they differ significantly in how learning updates are performed.

Key Idea:
Reinforce learns only from rewards collected after an episode ends, while Actor-Critic introduces a Critic that provides immediate feedback, making learning faster and more stable.

Table of Contents

• Introduction to Reinforcement Learning
• Core Elements of RL
• What is Reinforce?
• Understanding Policy Gradient
• Problems with Reinforce
• What is Actor-Critic?
• Understanding the Actor
• Understanding the Critic
• Advantage Function
• Mathematics Behind Actor-Critic
• Reinforce vs Actor-Critic Comparison
• Applications
• Final Thoughts

Introduction to Reinforcement Learning

Reinforcement Learning is inspired by behavioral psychology. An agent learns by interacting with an environment and observing the consequences of its actions.

The process follows a simple cycle:

1. The agent observes the current state.
2. The agent selects an action.
3. The environment responds with a reward.
4. The environment transitions into a new state.
5. The agent updates its strategy.

Over time, the agent tries to maximize cumulative rewards.

Expected Return

The total reward collected over time is called the return:

$$ G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + ... $$

Where:

• \(G_t\) = return at time step \(t\)
• \(R_t\) = reward
• \(\gamma\) = discount factor

The discount factor determines how much future rewards matter.

Core Elements of Reinforcement Learning

Concept	Description
Agent	The learner or decision maker
Environment	The world the agent interacts with
State	The current situation of the environment
Action	The decision taken by the agent
Reward	Feedback received from the environment
Policy	The strategy used by the agent

What is Reinforce?

Reinforce is one of the simplest policy gradient algorithms. Instead of learning value tables like Q-Learning, Reinforce directly learns the policy itself.

A policy is usually represented as:

$$ \pi_\theta(a|s) $$

This means:

• \(s\) = state
• \(a\) = action
• \(\theta\) = parameters of the policy network

The policy outputs probabilities for actions.

How Reinforce Works

1. Run a complete episode.
2. Collect rewards.
3. Compute total return.
4. Update the policy toward rewarding actions.

Important:
Reinforce waits until the episode ends before updating the model.

Understanding Policy Gradient

Policy gradient methods optimize policies directly using gradients.

The objective is:

$$ J(\theta) = \mathbb{E}[G_t] $$

We want to maximize expected rewards.

The gradient update becomes:

$$ \nabla J(\theta) = \mathbb{E}[\nabla \log \pi_\theta(a|s) G_t] $$

This equation tells us:

• If rewards are high, increase the probability of chosen actions.
• If rewards are low, reduce the probability.

Intuition Behind Log Probability

Suppose:

$$ \pi_\theta(a|s)=0.8 $$

Then:

$$ \log(0.8) \approx -0.223 $$

The gradient adjusts parameters depending on reward signals.

Problems with Reinforce

Although Reinforce is elegant, it has major limitations.

1. High Variance

Updates can fluctuate significantly because they depend on complete episodic returns.

Two identical actions may appear very different if total episode rewards differ.

2. Slow Learning

The algorithm waits until the episode finishes.

For long episodes, this becomes inefficient.

3. Credit Assignment Problem

It becomes difficult to determine which action caused success or failure.

Variance Formula

Variance measures instability:

$$ Var(X)=\mathbb{E}[(X-\mu)^2] $$

Higher variance means more unstable learning.

What is Actor-Critic?

Actor-Critic combines two ideas:

• Policy learning
• Value estimation

Instead of relying only on rewards at the end of an episode, Actor-Critic introduces a Critic that evaluates actions immediately.

The Two Components

Component	Purpose
Actor	Selects actions
Critic	Evaluates actions

Understanding the Actor

The Actor behaves similarly to Reinforce.

It maintains a policy:

$$ \pi_\theta(a|s) $$

The Actor decides which action to take.

Its goal is to maximize rewards.

Understanding the Critic

The Critic estimates how good the current state is.

This is done using the value function:

$$ V^\pi(s) $$

The value function estimates expected future rewards from a state.

Instead of waiting until the end of the episode, the Critic gives immediate feedback.

Core Improvement:
The Critic reduces uncertainty by estimating future rewards continuously.

Advantage Function

Actor-Critic often uses the Advantage Function.

The advantage tells us whether an action was better or worse than expected.

$$ A(s,a)=Q(s,a)-V(s) $$

Where:

• \(Q(s,a)\) = action-value function
• \(V(s)\) = state-value function

Positive advantage means the action was better than expected.

Negative advantage means the action was worse.

Mathematics Behind Actor-Critic

The Critic updates value estimates using Temporal Difference learning.

TD Error

$$ \delta = r + \gamma V(s') - V(s) $$

Where:

• \(r\) = immediate reward
• \(s'\) = next state
• \(\gamma\) = discount factor

The TD error measures how surprising the outcome was.

Example Calculation

Suppose:

• Reward = 5
• \(\gamma = 0.9\)
• \(V(s') = 8\)
• \(V(s)=10\)

Then:

$$ \delta = 5 + 0.9(8) - 10 $$ $$ \delta = 5 + 7.2 -10 $$ $$ \delta = 2.2 $$

A positive TD error means the outcome was better than expected.

Actor Update Rule

The Actor updates using:

$$ \nabla J(\theta)=\nabla \log \pi_\theta(a|s)\delta $$

Instead of using full episode returns, Actor-Critic uses TD error for immediate learning.

Critic Update Rule

The Critic minimizes prediction error:

$$ V(s) \leftarrow V(s)+\alpha\delta $$

Where:

• \(\alpha\) = learning rate
• \(\delta\) = TD error

Reinforce vs Actor-Critic Comparison

Feature	Reinforce	Actor-Critic
Learning Type	Policy Gradient	Policy + Value Learning
Variance	High	Lower
Learning Speed	Slow	Faster
Episode Dependency	Needs full episode	Step-by-step updates
Complexity	Simpler	More advanced
Scalability	Limited	Better for large problems

Why Lower Variance Matters

Stable learning is crucial in Reinforcement Learning.

High variance can cause:

• Oscillating performance
• Slow convergence
• Unstable policies
• Poor exploration

Actor-Critic reduces variance by using value estimates.

Bias-Variance Tradeoff

Machine learning often balances:

$$ Error = Bias^2 + Variance + Noise $$

Reinforce has low bias but high variance.

Actor-Critic introduces slightly more bias but greatly reduces variance.

Applications of Actor-Critic

Actor-Critic methods are widely used in modern AI systems.

• Robotics
• Autonomous driving
• Game playing
• Recommendation systems
• Financial trading
• Industrial optimization
• Resource allocation
• Smart control systems

Popular Actor-Critic Variants

Algorithm	Description
A2C	Advantage Actor-Critic
A3C	Asynchronous Advantage Actor-Critic
PPO	Proximal Policy Optimization
DDPG	Deep Deterministic Policy Gradient
SAC	Soft Actor-Critic

Why PPO Became Popular

PPO stabilizes Actor-Critic updates by limiting how drastically policies can change during training.

This prevents destructive updates and improves reliability.

Code Intuition (Pseudo Workflow)

for each step:

    action = actor(state)

    next_state, reward = environment(action)

    td_error = reward + gamma * V(next_state) - V(state)

    actor.update(td_error)

    critic.update(td_error)

    state = next_state

This demonstrates how Actor-Critic learns continuously after every step.

Final Thoughts

Reinforce is one of the best algorithms for understanding the foundations of policy optimization. It introduces the core concept of directly learning policies from rewards.

However, its reliance on complete episodic returns makes learning unstable and inefficient in complex environments.

Actor-Critic improves upon Reinforce by introducing a Critic that estimates value functions and provides immediate feedback. This reduces variance, speeds up learning, and makes the algorithm practical for real-world applications.

Final Takeaway:
Reinforce teaches the fundamentals of policy gradients, while Actor-Critic demonstrates how combining policy learning with value estimation creates faster and more stable reinforcement learning systems.

What Is Asymptotic Correctness? A Simple Guide for RL Beginners

Asymptotic Correctness in Reinforcement Learning – Complete Guide

Reinforcement Learning (RL) is one of the most exciting areas in artificial intelligence and machine learning. Unlike supervised learning, where models learn from labeled examples, reinforcement learning focuses on agents learning through interaction with environments.

An RL agent continuously makes decisions, receives rewards or penalties, and gradually improves its behavior over time.

However, one major question always arises:

Will the agent eventually learn the optimal strategy if given enough experience?

This exact question leads us to the concept of asymptotic correctness.

💡 Key Takeaways

• Asymptotic correctness means eventual convergence to the optimal policy.
• Exploration is essential for discovering optimal actions.
• Algorithms like Q-learning and SARSA can converge under proper conditions.
• Learning rates and discount factors strongly affect convergence.
• Bellman equations form the mathematical backbone of RL.
• Finite-time performance may still remain challenging.

---

Table of Contents

• 1. Introduction to Reinforcement Learning
• 2. Markov Decision Processes
• 3. What is Asymptotic Correctness?
• 4. Exploration vs Exploitation
• 5. Q-Learning
• 6. SARSA Algorithm
• 7. Bellman Equations
• 8. Convergence Theory
• 9. Mathematical Foundations
• 10. CLI Simulation Examples
• 11. Limitations
• 12. Real World Applications
• 13. Conclusion

---

1. Introduction to Reinforcement Learning

Reinforcement learning is inspired by how humans and animals learn from experience.

Imagine teaching a dog new tricks:

• Correct behavior → reward
• Incorrect behavior → no reward

Over time, the dog learns actions that maximize rewards.

RL agents work similarly.

Core Components of RL

Component	Description
Agent	The learner making decisions
Environment	The world the agent interacts with
State	Current situation of the environment
Action	Possible move taken by the agent
Reward	Feedback signal
Policy	Strategy for choosing actions

The objective is:

$$ \text{Maximize Expected Cumulative Reward} $$

---

2. Markov Decision Processes

Most reinforcement learning problems are modeled using a Markov Decision Process (MDP).

MDP Components

$$ MDP = (S, A, P, R, \gamma) $$ Where:

• \(S\) = Set of states
• \(A\) = Set of actions
• \(P\) = Transition probabilities
• \(R\) = Reward function
• \(\gamma\) = Discount factor

Transition Probability

The probability of moving from one state to another:

$$ P(s'|s,a) $$

This means:

Probability of reaching state \(s'\) after taking action \(a\) in state \(s\).

Discount Factor

The discount factor determines how future rewards are valued.

$$ 0 \leq \gamma \leq 1 $$

• \(\gamma \approx 0\): Focus on immediate rewards
• \(\gamma \approx 1\): Focus on long-term rewards

---

3. What is Asymptotic Correctness?

Asymptotic correctness refers to the long-term convergence behavior of an RL algorithm.

If:

$$ \pi_t \to \pi^* $$

as:

$$ t \to \infty $$

then the algorithm is asymptotically correct.

Where:

• \(\pi_t\) = learned policy at time \(t\)
• \(\pi^*\) = optimal policy

Simple Interpretation

Given infinite learning time:

• The agent explores enough.
• The agent gathers enough information.
• The agent eventually discovers the best possible strategy.

Maze Example

Imagine an agent navigating a maze:

• Initially, it randomly explores.
• Over time, it remembers good paths.
• Eventually, it consistently finds the shortest route.

This demonstrates convergence toward optimal behavior.

---

4. Exploration vs Exploitation

One of the most difficult problems in RL is balancing:

• Exploration → Trying new actions
• Exploitation → Using known good actions

Why Exploration Matters

Without exploration:

• The agent may never discover better strategies.
• The policy may become trapped in local optima.

Epsilon-Greedy Strategy

A common method is epsilon-greedy exploration.

$$ P(\text{explore}) = \epsilon $$ $$ P(\text{exploit}) = 1 - \epsilon $$

Decay Schedule

Typically:

$$ \epsilon_t \to 0 $$

as training progresses.

Interpretation

• Early training → more exploration
• Late training → more exploitation

---

5. Q-Learning

Q-learning is one of the most important reinforcement learning algorithms.

Core Idea

Instead of directly learning policies, Q-learning estimates:

$$ Q(s,a) $$

which represents:

Expected future reward for taking action \(a\) in state \(s\).

Q-Learning Update Rule

$$ Q(s,a) \leftarrow Q(s,a) + \alpha \left[r + \gamma \max_{a'}Q(s',a') - Q(s,a)\right] $$ Where:

• \(\alpha\) = learning rate
• \(r\) = reward
• \(\gamma\) = discount factor
• \(s'\) = next state

Mathematical Intuition

The update adjusts estimates toward:

$$ \text{Target} = r + \gamma \max_{a'}Q(s',a') $$

The difference:

$$ \delta = \text{Target} - Q(s,a) $$

is called the temporal difference error.

Why Q-Learning Converges

Under proper conditions:

• All state-action pairs are visited infinitely often.
• Learning rates decay properly.
• The environment satisfies MDP assumptions.

Then:

$$ Q_t(s,a) \to Q^*(s,a) $$

---

6. SARSA Algorithm

SARSA stands for:

$$ (State, Action, Reward, State, Action) $$

SARSA Update Rule

$$ Q(s,a) \leftarrow Q(s,a) + \alpha \left[r + \gamma Q(s',a') - Q(s,a)\right] $$

Difference Between SARSA and Q-Learning

Q-Learning	SARSA
Off-policy	On-policy
Uses max future reward	Uses actual chosen action
More aggressive learning	Safer exploration

SARSA Convergence

SARSA is asymptotically correct if:

• Exploration continues sufficiently.
• Learning rates decay properly.

---

7. Bellman Equations

Bellman equations are the foundation of reinforcement learning.

Bellman Optimality Equation

$$ V^*(s) = \max_a \sum_{s'} P(s'|s,a)\left[R(s,a,s') + \gamma V^*(s')\right] $$

Interpretation

The value of a state equals:

• Immediate reward
• Plus discounted future value

Recursive Nature

Bellman equations are recursive because:

$$ V(s) $$

depends on:

$$ V(s') $$

This recursive structure allows dynamic programming methods.

---

8. Convergence Theory

Convergence Definition

Convergence means:

$$ \lim_{t \to \infty} Q_t = Q^* $$

Conditions for Convergence

Condition	Purpose
Infinite exploration	Discover all actions
Decaying learning rate	Stabilize updates
Bounded rewards	Prevent divergence
Markov property	Ensure consistent transitions

Learning Rate Requirement

A common condition:

$$ \sum_{t=1}^{\infty} \alpha_t = \infty $$ and: $$ \sum_{t=1}^{\infty} \alpha_t^2 < \infty $$

These conditions ensure:

• Sufficient learning
• Eventual stability

---

9. Mathematical Foundations

Expected Return

$$ G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} $$

This represents cumulative discounted reward.

Interpretation

• Immediate rewards matter more.
• Future rewards gradually decay.

Contraction Mapping

Bellman operators are contraction mappings:

$$ ||T(V_1)-T(V_2)|| \leq \gamma ||V_1 - V_2|| $$

Since:

$$ \gamma < 1 $$

Repeated application guarantees convergence.

Fixed Point Concept

Optimal values satisfy:

$$ T(V^*) = V^* $$

Meaning:

• The optimal value function becomes stable.
• Further updates no longer change it.

---

10. CLI Simulation Examples

Python Q-Learning Example

Q[state][action] = Q[state][action] + alpha * (
reward + gamma * max(Q[next_state]) - Q[state][action]
)

CLI Output

$ python train_agent.py

Episode 1 Reward: -10
Episode 50 Reward: 20
Episode 100 Reward: 40
Episode 500 Reward: 100

Optimal policy discovered.

Policy Evaluation Output

$ python evaluate.py

Average Reward: 98.7
Convergence achieved.

TensorFlow RL Example

import tensorflow as tf

model.compile(
optimizer='adam',
loss='mse'
)

---

11. Limitations of Asymptotic Correctness

1. Infinite Time Assumption

Asymptotic guarantees assume:

$$ t \to \infty $$

Real systems rarely have infinite training time.

2. Large State Spaces

Modern environments may contain:

$$ |S| \gg 10^6 $$

Exploring all states becomes difficult.

3. Expensive Exploration

Exploration may cause:

• Financial losses
• Unsafe robotic behavior
• Poor user experiences

4. Non-Stationary Environments

If environments change over time:

$$ P_t(s'|s,a) \neq P_{t+1}(s'|s,a) $$

Convergence guarantees may fail.

---

12. Real World Applications

Robotics

Robots learn navigation and manipulation strategies.

Game AI

Systems like AlphaGo rely heavily on reinforcement learning.

Recommendation Systems

Streaming platforms optimize recommendations using reward feedback.

Autonomous Vehicles

Cars learn safe driving strategies through simulations.

Finance

Trading agents attempt to maximize long-term returns.

---

13. Conclusion

Asymptotic correctness is one of the most important theoretical guarantees in reinforcement learning. It assures us that under appropriate conditions, RL algorithms can eventually discover optimal behavior.

Algorithms like:

• Q-learning
• SARSA
• Temporal Difference methods

demonstrate how agents can iteratively improve through repeated interactions with environments.

However, practical reinforcement learning is not only about eventual convergence.

Real-world systems must also address:

• Finite-time efficiency
• Safe exploration
• Scalability
• Computational constraints

The study of asymptotic correctness provides the mathematical foundation for understanding why reinforcement learning works, while modern research focuses on making convergence faster, safer, and more efficient.

🎯 Final Summary

• Asymptotic correctness means eventual convergence to optimal policies.
• Exploration is essential for learning.
• Bellman equations drive RL updates.
• Q-learning and SARSA converge under suitable conditions.
• Learning rates and discount factors strongly influence stability.
• Practical RL still faces major finite-time challenges.