Impact of Regularization on Decision Tree Regression

🌳 Decision Tree Regularization – A Story of Simplicity vs Complexity

Imagine you're trying to draw a smooth curve through a messy set of points. Do you draw a simple line… or a super detailed zig-zag that passes through every point?

That exact dilemma is called: Underfitting vs Overfitting

This blog walks you through it using decision trees—step by step.

---

📚 Table of Contents

• Data Generation
• Two Models
• Math Behind It
• Code Example
• CLI Output
• Understanding the Plot
• Regularization Explained
• Key Takeaways
• Related Articles

---

🎲 Step 1: Generating Data

X = np.sort(5 * rng.rand(80, 1), axis=0) y = np.sin(X).ravel() y[::5] += 3 * (0.5 - rng.rand(16))

This creates:

• 80 random points between 0 and 5
• A sine curve as the base pattern
• Noise added every 5th point

👉 Real-world data is never clean—noise simulates reality.

---

⚙️ Step 2: Two Competing Models

Model	Settings	Behavior
Model 1	max_depth=2	Simple (Underfits)
Model 2	max_depth=5, min_samples_leaf=10	Complex (Balanced)

regr_1 = DecisionTreeRegressor(max_depth=2) regr_2 = DecisionTreeRegressor(max_depth=5, min_samples_leaf=10) regr_1.fit(X, y) regr_2.fit(X, y)

---

📐 The Math (Made Easy)

1. Model Error

\[ Error = Bias^2 + Variance + Noise \]

Simple Meaning:

• Bias → Too simple (misses pattern)
• Variance → Too complex (fits noise)
• Noise → Randomness in data

👉 Good model = Balance between bias and variance

2. Tree Depth Effect

\[ Depth \uparrow \Rightarrow Variance \uparrow \]

\[ Depth \downarrow \Rightarrow Bias \uparrow \]

Meaning:

• Deeper trees → more flexible → risk overfitting
• Shallow trees → more rigid → risk underfitting

3. Leaf Constraint

\[ Leaf\ Size \uparrow \Rightarrow Smoother\ Model \]

This prevents tiny splits that memorize noise.

---

💻 Step 3: Predictions

X_test = np.arange(0.0, 5.0, 0.01)[:, np.newaxis] y_1 = regr_1.predict(X_test) y_2 = regr_2.predict(X_test)

---

🖥️ CLI Output (Conceptual)

Click to Expand

Model 1 (Depth=2):
- Smooth curve
- Misses fluctuations

Model 2 (Depth=5):

* Follows data closely
* Captures more detail

  

---

📊 Understanding the Plot

• Orange dots → Actual noisy data
• Blue line → Shallow tree
• Green line → Deeper tree

🔵 Shallow Tree (max_depth=2)

• Captures overall trend
• Misses detail
• Underfitting

🟢 Deeper Tree (max_depth=5)

• Captures more patterns
• More flexible
• Risk of overfitting

👉 The goal is NOT perfect fit—it's generalization.

---

🛡️ What is Regularization?

Regularization controls how complex your model becomes.

Key Techniques:

• max_depth → limits tree size
• min_samples_leaf → prevents tiny splits

Think of it like:

“Don’t let the model memorize—force it to learn patterns.”

---

💡 Key Takeaways

• Shallow trees = simple but may underfit
• Deep trees = powerful but may overfit
• Regularization balances both
• Math helps explain model behavior clearly

---

🎯 Final Insight

A perfect model is not the one that fits the training data best…

It’s the one that performs best on unseen data.