Why the Sigmoid Function is NOT a True Probability Function

The sigmoid function is widely used in machine learning, especially in classification tasks, and is often associated with probabilities. It maps real numbers into a range between 0 and 1, which makes it look like a probability — but that’s not the full story.

📚 Table of Contents

• What is the Sigmoid Function?
• Sigmoid vs Probability
• 1. Not a True Probability Distribution
• 2. Output Can Be Misleading
• 3. Not Calibrated
• 4. Ignores Competing Outcomes
• Code Example
• CLI Output Example
• Key Takeaways
• Related Articles

---

What is the Sigmoid Function?

The sigmoid function, often written as:

σ(x) = 1 / (1 + e^-x)

Transforms any real value into a number between 0 and 1.

🔍 Why is this important?

This transformation is useful in machine learning because models often output values in the range (-∞, +∞), and sigmoid compresses them into a bounded range that resembles probabilities.

Behavior of Sigmoid

• Large negative → output ≈ 0
• Zero → output = 0.5
• Large positive → output ≈ 1

Sigmoid and Probabilities

Yes — sigmoid outputs look like probabilities. But there’s a critical distinction:

Output in [0,1] ≠ Valid Probability Distribution

1. Sigmoid is NOT a True Probability Distribution

A true probability function must satisfy:

• All probabilities ≥ 0
• Total probability = 1

⚠️ Problem with Sigmoid

Sigmoid gives probability of a single class but does not inherently ensure that:

P(class A) + P(class B) = 1

This only works if you explicitly define:

P(B) = 1 - P(A)

2. Sigmoid Output Can Be Misleading

Sigmoid has uneven sensitivity:

• Very sensitive near 0
• Very insensitive at extremes

📉 Why this matters

Small changes in input can drastically change predictions near 0, but huge changes barely matter at extremes.

3. Sigmoid is NOT Calibrated

A calibrated model means:

Predicted 70% → Happens ~70% of time

⚠️ Reality

Sigmoid outputs are often:

• Overconfident
• Underconfident

Calibration techniques:

• Platt Scaling
• Isotonic Regression

4. Sigmoid Ignores Other Outcomes

Sigmoid works independently per class.

For multiple classes, we use:

Softmax Function

📊 Why Softmax is better

• Considers all classes together
• Ensures probabilities sum to 1

💻 Code Example

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

values = [-10, -1, 0, 1, 10]

for v in values:
    print(f"x={v}, sigmoid={sigmoid(v):.4f}")

🖥 CLI Output Example

$ python sigmoid_demo.py

x=-10, sigmoid=0.0000
x=-1,  sigmoid=0.2689
x=0,   sigmoid=0.5000
x=1,   sigmoid=0.7311
x=10,  sigmoid=1.0000

💡 Key Takeaways

• Sigmoid outputs are NOT true probabilities
• They don’t enforce total probability = 1
• They are sensitive to scaling
• They require calibration for real-world use
• Softmax is better for multi-class problems

🔗 Related Articles

• Why the Sigmoid Function is Considered a Probability Function
• Probability Mass Function (PMF) vs Probability Density Function (PDF)
• Simple Explanation of Sigmoid Function
• Comparing PDF and CDF

📌 Final Thought

Sigmoid is a powerful transformation tool — but not a complete probability model. Understanding this nuance separates surface-level ML usage from deeper mastery.