Reducing Overfitting in Decision Tree Regression by Limiting Depth

Decision Tree Regression: Controlling Overfitting with Max Depth

📚 Table of Contents

• Problem Overview
• Understanding the Plot
• What is Overfitting?
• Mathematics Behind Decision Trees
• Why Depth = 2 Works
• Code Example
• CLI Output
• Key Takeaways

---

Problem Overview

We aim to predict housing prices (MEDV) using the percentage of lower-income population (LSTAT).

This is a classic regression problem where:

\[ \text{MEDV} = f(\text{LSTAT}) \]

A Decision Tree Regressor is used to approximate this function.

💡 The goal is not just accuracy on training data—but generalization to unseen data.

---

Understanding the Plot

1. Scatter Plot

Each blue dot represents a real observation:

• X-axis: LSTAT (lower-income %)
• Y-axis: MEDV (house price)

2. Prediction Line

The black line is the model prediction.

🔍 Why is the line not smooth?

Decision trees do not create smooth curves. Instead, they divide the data into regions and assign a constant value to each region.

---

What is Overfitting?

Overfitting happens when a model learns noise instead of the underlying pattern.

With depth = 5:

• Too many splits
• Captures noise
• Poor generalization

Bias-Variance Tradeoff

\[ \text{Total Error} = \text{Bias}^2 + \text{Variance} + \text{Noise} \]

Deep trees → Low bias, High variance Shallow trees → High bias, Low variance

---

Mathematics Behind Decision Trees

1. Mean Squared Error (MSE)

\[ MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \]

This measures how far predictions are from actual values.

2. Splitting Criterion

A tree chooses splits that minimize MSE:

\[ \text{Best Split} = \arg\min \left( MSE_{left} + MSE_{right} \right) \]

3. Prediction Rule

Each leaf node predicts:

\[ \hat{y} = \frac{1}{n} \sum y_i \]

📘 Expand: Why constant predictions?

Decision trees average values inside each region. That’s why predictions appear as horizontal lines.

---

Why Reducing Depth to 2 Works

1. Fewer Splits

Only the most important patterns are captured.

2. Better Generalization

The model ignores noise and focuses on trends.

3. Simpler Model

A depth-2 tree creates at most:

\[ 2^2 = 4 \text{ leaf nodes} \]

This limits complexity significantly.

🎯 A simpler model often performs better on new data.

---

Code Example

from sklearn.tree import DecisionTreeRegressor
import numpy as np

X = data['LSTAT'].values.reshape(-1, 1)
y = data['MEDV']

model = DecisionTreeRegressor(max_depth=2)
model.fit(X, y)

predictions = model.predict(X)

---

CLI Output Example

$ python train_model.py

Loading dataset...
Training Decision Tree (depth=2)...

Training complete!
MSE: 18.42
Model complexity reduced
Overfitting minimized

---

Key Takeaways

• Decision trees create piecewise constant predictions
• Deep trees overfit, shallow trees generalize better
• max_depth is a powerful regularization tool
• Bias-variance tradeoff is key to model performance

---

Conclusion

Reducing the depth of a Decision Tree is one of the simplest yet most effective ways to combat overfitting.

By limiting complexity, we ensure the model captures meaningful patterns rather than noise—leading to better real-world performance.