How the Similarity Formula Works in XGBoost Trees

XGBoost Similarity Formula

📚 Table of Contents

• Core Idea
• Why Do We Need It?
• The Formula
• What are G and H?
• Simple Intuition
• How Splitting Works
• Numerical Example
• Code Example
• CLI Output
• Key Takeaways

---

🧠 Core Idea

XGBoost builds trees step by step. At each step, it asks:

💡 “If I split the data here, does my prediction improve?”

The similarity formula helps answer this question.

---

❓ Why Do We Need It?

In normal decision trees:

• We use Gini or entropy

But in XGBoost:

• We optimize a loss function (like error)
• We need a smarter way to measure improvement

💡 So instead of “purity”, XGBoost measures “how much error reduces”

---

📐 The Similarity Formula

Similarity = G² / (H + λ)

Where:

• G = total gradient
• H = total hessian
• λ = regularization (controls overfitting)

---

🔍 What are G and H?

Gradient (G)

Tells how wrong the prediction is.

Hessian (H)

Tells how confident we are about that error.

💡 Simple:
Gradient = direction to improve
Hessian = confidence in that direction

---

💡 Intuition Behind Formula

Think of similarity as:

💡 “How strong is this group of data?”

• High G → big error → potential improvement
• High H → stable information
• λ → prevents overfitting

---

🌳 How XGBoost Chooses Splits

1. Try a split
2. Calculate similarity for left node
3. Calculate similarity for right node
4. Compare with parent node

Gain formula:

Gain = Left + Right - Parent

💡 Highest gain = best split

---

📊 Simple Example

Assume:

G = 10
H = 5
λ = 1

Similarity:

= 10² / (5 + 1)
= 100 / 6
≈ 16.67

Higher value = better node

---

💻 Code Example

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2
)

model = xgb.XGBClassifier()
model.fit(X_train, y_train)

print(model.score(X_test, y_test))

---

🖥 CLI Output

Accuracy: 0.96

---

🎯 Key Takeaways

✔ XGBoost uses gradients instead of simple counts ✔ Similarity measures node quality ✔ Gain decides best split ✔ λ prevents overfitting ✔ Better splits → better predictions

---

📚 Related Articles

• Tree Method in XGBoost

---

🚀 Final Thought

The similarity formula is the brain of XGBoost trees. Once you understand it, the whole algorithm becomes much easier to follow.