🌸 Decision Tree Classifier on Iris Dataset (With & Without Pruning)
This guide walks you through one of the most fundamental machine learning problems using the Iris dataset. The goal is not just to build a model—but to deeply understand how pruning affects decision trees.
📚 Table of Contents
- Introduction
- Understanding the Dataset
- Decision Tree Basics
- Mathematics Behind Decision Trees
- Baseline Model
- Pruning Explained
- Model Comparison
- Interactive Sections
- Key Takeaways
- Related Articles
🚀 Introduction
The Iris dataset is one of the most famous datasets in machine learning. It contains 150 samples of flowers belonging to three species. Each sample has four features:
- Sepal Length
- Sepal Width
- Petal Length
- Petal Width
The task is to classify the species using a Decision Tree Classifier.
📊 Understanding the Dataset
The dataset is perfectly balanced:
- 50 samples per species
- 3 classes total
This balance is one reason why decision trees perform exceptionally well here.
🌳 Decision Tree Basics
A Decision Tree works by splitting data based on feature thresholds. Each split is chosen to maximize class separation.
Example rule:
if petal_length < 2.5 → Setosa
🧠 Mathematics Behind Decision Trees
1. Gini Impurity
\[ Gini = 1 - \sum_{i=1}^{n} p_i^2 \]
Where \( p_i \) is the probability of class \( i \).
Explanation:
Gini measures how mixed the classes are. Lower is better.
2. Entropy
\[ Entropy = -\sum_{i=1}^{n} p_i \log_2(p_i) \]
This measures randomness in the dataset.
3. Information Gain
\[ IG = Entropy(parent) - \sum (weight \times Entropy(child)) \]
This tells us how much a split improves classification.
📌 Baseline Model (No Pruning)
Code Example
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier()
model.fit(X_train, y_train)
CLI Output
View Output
Train Accuracy: 100% Test Accuracy: 100% Tree Depth: 6 Leaves: 10
This indicates perfect classification.
✂️ Pruning Explained
Pruning removes unnecessary branches to reduce overfitting.
Cost Complexity Formula
\[ R_\alpha(T) = R(T) + \alpha |T| \]
- \(R(T)\): Error of tree
- \(|T|\): Number of leaves
- \(\alpha\): Regularization parameter
📉 Pruning Results
ccp_alpha = 0.00924
Train Accuracy: 99% Test Accuracy: 100% Depth: 6 Leaves: 8
ccp_alpha = 0.01270
Train Accuracy: 97.14% Test Accuracy: 100% Depth: 4 Leaves: 6
ccp_alpha = 0.01847
Train Accuracy: 94.29% Test Accuracy: 100% Depth: 3 Leaves: 4
ccp_alpha = 0.02706
Train Accuracy: 94.29% Test Accuracy: 97.78% Depth: 2 Leaves: 3
ccp_alpha = 0.25029
Train Accuracy: 64.76% Test Accuracy: 71.11% Depth: 1 Leaves: 2
ccp_alpha = 0.31211
Train Accuracy: 35.24% Test Accuracy: 28.89% Depth: 0 Leaves: 1
⚖️ Model Comparison
| ccp_alpha | Depth | Leaves | Train Accuracy | Test Accuracy |
|---|---|---|---|---|
| 0.00000 | 6 | 10 | 100% | 100% |
| 0.01270 | 4 | 6 | 97% | 100% |
| 0.02706 | 2 | 3 | 94% | 97% |
| 0.31211 | 0 | 1 | 35% | 28% |
🧩 Interactive Learning
Try adjusting ccp_alpha and observe how:
- Tree depth changes
- Accuracy changes
- Overfitting reduces
💡 Key Takeaways
- Decision Trees can overfit easily
- Pruning reduces complexity
- Too much pruning harms performance
- For Iris dataset, pruning was unnecessary
🎯 Final Conclusion
In this case, the simplest answer was the best: no pruning. The dataset is clean and well-separated, so a full tree performs perfectly.
However, in real-world datasets, pruning is often essential.
