Understanding DQN and Fitted Q Iteration in Reinforcement Learning
Reinforcement Learning (RL) is one of the most exciting areas of Artificial Intelligence. Instead of explicitly telling a machine what to do, reinforcement learning allows an agent to learn through interaction with an environment.
Think of it like teaching a child how to ride a bicycle. The child tries, falls, adjusts balance, learns from mistakes, and eventually becomes better. RL works in a very similar way.
Reinforcement learning is about learning through rewards and penalties.
Table of Contents
What is Reinforcement Learning?
Reinforcement Learning is a branch of Machine Learning where an agent learns by interacting with an environment.
The core components are:
- Agent → The learner or decision-maker
- Environment → The world the agent interacts with
- Action → What the agent does
- Reward → Feedback from the environment
- State → Current situation of the environment
The goal of the agent is simple:
Maximize cumulative rewards over time.
Reward Mathematics
The total reward is often represented as:
$$ R_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} $$Where:
- \(R_t\) = Total future reward
- \(r_t\) = Immediate reward
- \(\gamma\) = Discount factor
The discount factor controls how much future rewards matter.
Understanding Q-Learning
Q-learning is one of the most popular reinforcement learning algorithms.
It teaches an agent which action is best in a particular state.
The algorithm stores values in something called a Q-table.
| State | Action | Q-value |
|---|---|---|
| S1 | Left | 0.4 |
| S1 | Right | 0.9 |
The higher the Q-value, the better the action.
Q-values estimate how good an action is in a given state.
Q-Learning Mathematics
The Q-learning update rule is:
Where:
- \(Q(s,a)\) = Current Q-value
- \(\alpha\) = Learning rate
- \(r\) = Reward
- \(\gamma\) = Discount factor
- \(s'\) = Next state
This equation is the heart of Q-learning.
The agent updates its knowledge after every action.
Example
Suppose:
- Current Q-value = 2
- Reward = 5
- Maximum future Q-value = 8
- Learning rate = 0.1
- Discount factor = 0.9
Then:
The updated Q-value becomes 3.02.
Difficulties With Traditional Q-Learning
Q-learning works well for small problems.
However, it struggles when:
- The state space becomes huge
- The action space becomes large
- The environment is continuous
- Memory requirements explode
For example:
- Chess has enormous state combinations
- Video games contain millions of screen states
- Robotics environments are highly dynamic
This is where DQN becomes important.
What is Deep Q Network (DQN)?
Deep Q Networks combine:
- Q-learning
- Deep Neural Networks
Instead of storing Q-values inside a table, DQN uses a neural network to predict Q-values.
Traditional Q-learning uses a table.
DQN uses a neural network.
Why Neural Networks?
Neural networks are excellent at approximating functions.
The network takes a state as input and outputs predicted Q-values for all possible actions.
DQN Approximation
$$ Q(s,a) \approx Q(s,a;\theta) $$Where:
- \(\theta\) = Neural network weights
How DQN Works
- Observe current state
- Select an action
- Perform the action
- Receive reward
- Store experience
- Train neural network
- Repeat
Experience Replay
One of the most important innovations in DQN is Experience Replay.
The agent stores experiences:
$$ (state, action, reward, next\ state) $$These experiences are replayed later during training.
Why Is Experience Replay Useful?
- Breaks correlation between experiences
- Improves learning stability
- Increases data efficiency
- Prevents catastrophic forgetting
Replay Memory Sampling
Suppose replay memory size:
$$ N = 100000 $$Mini-batch size:
$$ B = 32 $$Random sampling improves generalization.
Target Networks
DQN also uses target networks.
Without target networks, learning becomes unstable.
The target network is updated periodically instead of continuously.
This stabilizes training significantly.
Neural Network Architecture in DQN
A DQN neural network often contains:
- Input layer
- Hidden layers
- Output layer
For image-based environments:
- Convolutional Neural Networks (CNNs) are commonly used
For simpler environments:
- Fully connected dense layers are enough
DQN Loss Function
The network minimizes prediction error.
What is Fitted Q Iteration?
Fitted Q Iteration is another reinforcement learning method built on Q-learning concepts.
Unlike DQN, Fitted Q Iteration uses batch learning.
Instead of learning continuously in real time, it:
- Collects experiences
- Builds a dataset
- Trains a regression model
- Updates the Q-function iteratively
Common Models Used
- Decision Trees
- Random Forests
- Linear Regression
- Gradient Boosting
- Neural Networks
Fitted Q Iteration learns from batches of experiences instead of continuous updates.
How Fitted Q Iteration Works
- Collect environment experiences
- Create training dataset
- Estimate target Q-values
- Train regression model
- Repeat until convergence
Target Computation
$$ y_i = r_i + \gamma \max_{a'} Q(s'_i,a') $$The model repeatedly improves predictions.
Advantages of Fitted Q Iteration
- Works well with offline datasets
- Stable training process
- Good for limited interaction environments
- Can use classical ML algorithms
Limitations of Fitted Q Iteration
- Slower than DQN
- Not ideal for real-time learning
- Requires batch retraining
- Computationally expensive for huge datasets
DQN vs Fitted Q Iteration
| Feature | DQN | Fitted Q Iteration |
|---|---|---|
| Learning Style | Online | Batch |
| Function Approximation | Neural Networks | Regression Methods |
| Real-Time Learning | Yes | No |
| Training Speed | Fast | Slower |
| Offline Dataset Support | Moderate | Excellent |
| Complex Environments | Excellent | Good |
Real World Applications
DQN Applications
- Atari video games
- Robotics
- Autonomous vehicles
- Recommendation systems
- Financial trading
Fitted Q Iteration Applications
- Medical treatment planning
- Offline robotics training
- Industrial optimization
- Energy management systems
Why DeepMind's DQN Was Revolutionary
DeepMind demonstrated that DQN could play Atari games directly from pixels.
This was groundbreaking because:
- No handcrafted features were needed
- The system learned automatically
- The same algorithm worked across multiple games
Bellman Equation
$$ Q^*(s,a) = \mathbb{E}[r + \gamma \max_{a'}Q^*(s',a')] $$This equation forms the theoretical backbone of many RL algorithms.
Simple Q-Learning Example
import numpy as np
q_table = np.zeros((5,2))
learning_rate = 0.1
discount = 0.9
state = 0
action = 1
reward = 5
next_state = 1
q_table[state, action] = q_table[state, action] + learning_rate * (
reward + discount * np.max(q_table[next_state]) - q_table[state, action]
)
print(q_table)
Simple DQN Workflow Example
state = environment.reset()
while True:
action = model.predict(state)
next_state, reward = environment.step(action)
replay_memory.append((state, action, reward, next_state))
train_network(replay_memory)
state = next_state
Future of Reinforcement Learning
Reinforcement learning is evolving rapidly.
Modern advancements include:
- Double DQN
- Dueling DQN
- Rainbow DQN
- Policy Gradient Methods
- Actor-Critic Algorithms
- Deep Deterministic Policy Gradient (DDPG)
- Proximal Policy Optimization (PPO)
These methods improve:
- Stability
- Sample efficiency
- Exploration
- Scalability
Final Thoughts
Q-learning laid the foundation for modern reinforcement learning.
DQN extended Q-learning using neural networks, allowing agents to solve complex high-dimensional problems.
Fitted Q Iteration introduced a powerful batch-learning approach that works well with offline datasets and structured learning.
Both methods remain extremely important in modern AI systems.
Q-learning teaches action values.
DQN scales Q-learning using neural networks.
Fitted Q Iteration learns from batches of past experiences.
No comments:
Post a Comment