Sigmoid vs Tanh: Understanding Key Activation Functions in Neural Networks

Sigmoid vs Tanh Activation Functions (Complete Guide)

📌 Table of Contents

• Introduction
• Sigmoid Function
• Tanh Function
• Mathematical Understanding
• Comparison
• When to Use Each
• Modern Perspective (ReLU)
• Key Takeaways
• Related Articles

---

Introduction

Activation functions are the backbone of neural networks. Without them, a neural network would behave like a simple linear model, no matter how many layers it has.

💡 Activation functions introduce non-linearity, allowing neural networks to learn complex patterns.

---

Sigmoid Function

The Sigmoid (logistic) function converts any input into a probability value between 0 and 1.

$$ \sigma(x) = \frac{1}{1 + e^{-x}} $$

📊 Interpretation

• If \( x \to +\infty \), then output → 1
• If \( x \to -\infty \), then output → 0

• Output range: (0,1)
• Used in binary classification
• Suffers from vanishing gradient

💻 Code Example

import numpy as np def sigmoid(x): return 1 / (1 + np.exp(-x))

---

Tanh Function

The Tanh function expands the output range to include negative values.

$$ \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} $$

📊 Interpretation

• If \( x \to +\infty \), output → 1
• If \( x \to -\infty \), output → -1

• Output range: (-1,1)
• Zero-centered
• Better gradient flow than Sigmoid

💻 Code Example

import numpy as np def tanh(x): return np.tanh(x)

---

📊 Mathematical Deep Dive

Derivative of Sigmoid

$$ \sigma'(x) = \sigma(x)(1 - \sigma(x)) $$

This derivative becomes very small when \( \sigma(x) \) is near 0 or 1 → causing vanishing gradients.

Derivative of Tanh

$$ \tanh'(x) = 1 - \tanh^2(x) $$

This maintains stronger gradients near zero compared to Sigmoid.

Vanishing Gradient Concept

Gradient-based learning depends on:

$$ \frac{\partial L}{\partial w} $$

If gradients shrink → learning slows dramatically.

---

Comparison

Feature	Sigmoid	Tanh
Range	(0,1)	(-1,1)
Zero-centered	No	Yes
Gradient	Weak	Stronger
Usage	Output layer	Hidden layers

---

When to Use Each

• Sigmoid: Binary classification, probabilities
• Tanh: Hidden layers, faster convergence

---

Modern Perspective (ReLU)

Today, ReLU is preferred:

$$ f(x) = \max(0, x) $$

It avoids vanishing gradients for positive values.

💡 Sigmoid & Tanh are still important for understanding neural networks.

---

🎯 Key Takeaways

• Sigmoid outputs probabilities
• Tanh is zero-centered
• Both suffer from vanishing gradients
• ReLU is modern default

---

Conclusion

Sigmoid and Tanh are foundational activation functions that shaped modern deep learning. Understanding their mathematical behavior provides insight into how neural networks learn.

Why the Sigmoid Function is NOT a True Probability Function

The sigmoid function is widely used in machine learning, especially in classification tasks, and is often associated with probabilities. It maps real numbers into a range between 0 and 1, which makes it look like a probability — but that’s not the full story.

📚 Table of Contents

• What is the Sigmoid Function?
• Sigmoid vs Probability
• 1. Not a True Probability Distribution
• 2. Output Can Be Misleading
• 3. Not Calibrated
• 4. Ignores Competing Outcomes
• Code Example
• CLI Output Example
• Key Takeaways
• Related Articles

---

What is the Sigmoid Function?

The sigmoid function, often written as:

σ(x) = 1 / (1 + e^-x)

Transforms any real value into a number between 0 and 1.

🔍 Why is this important?

This transformation is useful in machine learning because models often output values in the range (-∞, +∞), and sigmoid compresses them into a bounded range that resembles probabilities.

Behavior of Sigmoid

• Large negative → output ≈ 0
• Zero → output = 0.5
• Large positive → output ≈ 1

Sigmoid and Probabilities

Yes — sigmoid outputs look like probabilities. But there’s a critical distinction:

Output in [0,1] ≠ Valid Probability Distribution

1. Sigmoid is NOT a True Probability Distribution

A true probability function must satisfy:

• All probabilities ≥ 0
• Total probability = 1

⚠️ Problem with Sigmoid

Sigmoid gives probability of a single class but does not inherently ensure that:

P(class A) + P(class B) = 1

This only works if you explicitly define:

P(B) = 1 - P(A)

2. Sigmoid Output Can Be Misleading

Sigmoid has uneven sensitivity:

• Very sensitive near 0
• Very insensitive at extremes

📉 Why this matters

Small changes in input can drastically change predictions near 0, but huge changes barely matter at extremes.

3. Sigmoid is NOT Calibrated

A calibrated model means:

Predicted 70% → Happens ~70% of time

⚠️ Reality

Sigmoid outputs are often:

• Overconfident
• Underconfident

Calibration techniques:

• Platt Scaling
• Isotonic Regression

4. Sigmoid Ignores Other Outcomes

Sigmoid works independently per class.

For multiple classes, we use:

Softmax Function

📊 Why Softmax is better

• Considers all classes together
• Ensures probabilities sum to 1

💻 Code Example

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

values = [-10, -1, 0, 1, 10]

for v in values:
    print(f"x={v}, sigmoid={sigmoid(v):.4f}")

🖥 CLI Output Example

$ python sigmoid_demo.py

x=-10, sigmoid=0.0000
x=-1,  sigmoid=0.2689
x=0,   sigmoid=0.5000
x=1,   sigmoid=0.7311
x=10,  sigmoid=1.0000

💡 Key Takeaways

• Sigmoid outputs are NOT true probabilities
• They don’t enforce total probability = 1
• They are sensitive to scaling
• They require calibration for real-world use
• Softmax is better for multi-class problems

🔗 Related Articles

• Why the Sigmoid Function is Considered a Probability Function
• Probability Mass Function (PMF) vs Probability Density Function (PDF)
• Simple Explanation of Sigmoid Function
• Comparing PDF and CDF

📌 Final Thought

Sigmoid is a powerful transformation tool — but not a complete probability model. Understanding this nuance separates surface-level ML usage from deeper mastery.

Implementing a Single-Layer Perceptron for AND Logic Gate Simulation

🧠 Building an AND Gate Using a Simple Neuron

Let’s build one of the simplest forms of intelligence—a neuron that behaves exactly like an AND logic gate.

This is the foundation of neural networks.

---

📚 Table of Contents

• Dataset
• Perceptron Model
• Math Explained Simply
• Training Process
• Code Example
• CLI Output
• Results
• Key Takeaways
• Related Articles

---

📊 Dataset

Input (x₁, x₂)	Output
(0, 0)	0
(0, 1)	0
(1, 0)	0
(1, 1)	1

---

⚙️ Perceptron Model

A perceptron works by calculating a weighted sum:

\[ z = w_1 x_1 + w_2 x_2 + b \]

Then applying an activation function:

\[ y = \begin{cases} 1 & \text{if } z > 0 \\ 0 & \text{otherwise} \end{cases} \]

👉 If the weighted sum crosses a threshold → output = 1

---

📐 Understanding the Math (Easy)

Why AND Works Linearly

The AND gate is linearly separable.

We can draw a line:

\[ w_1 x_1 + w_2 x_2 + b = 0 \]

That separates:

• (1,1) → one side
• All others → other side

Error Calculation

\[ Error = y_{true} - y_{pred} \]

Weight Update Rule

\[ w = w + \eta \cdot Error \cdot x \]

\[ b = b + \eta \cdot Error \]

Where:

• \(\eta\) = learning rate

👉 The model adjusts itself when it makes mistakes.

---

🔄 Training Process

• Loop through dataset
• Predict output
• Calculate error
• Update weights
• Repeat until error is minimized

---

💻 Code Example

import numpy as np X = np.array([[0,0],[0,1],[1,0],[1,1]]) y = np.array([0,0,0,1]) weights = np.random.rand(2) bias = np.random.rand() lr = 0.1 def step(z): return 1 if z > 0 else 0 for epoch in range(100): for i in range(len(X)): z = np.dot(X[i], weights) + bias pred = step(z) error = y[i] - pred ``` weights += lr * error * X[i] bias += lr * error ``` print(weights, bias)

---

🖥️ CLI Output

Click to Expand

Weights: [0.8, 0.7]
Bias: -1.1

---

✅ Results

Model Predictions

(0,0) → 0
(0,1) → 0
(1,0) → 0
(1,1) → 1

🎉 The neuron successfully learned the AND gate!

---

💡 Key Takeaways

• Perceptron = simplest neural model
• AND gate is linearly separable
• Learning happens via error correction
• This is the foundation of deep learning

---

🎯 Final Thought

If a machine can learn AND, it can learn logic. If it can learn logic… it can learn the world.

One-vs-One (OvO) vs. One-vs-Rest (OvR) in Multiclass Classification: A Simple Guide

When building machine learning models for **multiclass classification**, there are two common approaches for handling problems where the output has more than two classes: **One-vs-One (OvO)** and **One-vs-Rest (OvR)**. These methods allow binary classifiers (such as support vector machines or logistic regression) to handle multiclass problems.

Let's break down **OvO** and **OvR** in simple terms, compare the two, and see when to use each approach.

---

### What is One-vs-Rest (OvR)?

#### How it works:

- **One-vs-Rest** (also called **One-vs-All** or OvA) is a strategy where we train a separate binary classifier for each class. Each binary classifier tries to distinguish **one class** from **all other classes**.

  

For example, in a classification problem with 3 classes (let's say **A**, **B**, and **C**):

- One classifier will predict **"Class A vs not Class A"**.

- Another classifier will predict **"Class B vs not Class B"**.

- A third classifier will predict **"Class C vs not Class C"**.

#### Predictions:

- During prediction, all classifiers run on the input data, and the class with the **highest confidence score** is chosen as the final output.

#### Advantages of OvR:

- **Scalability**: It scales well when the number of classes is large, especially with efficient classifiers like logistic regression.

- **Simplicity**: It's straightforward to implement and understand, since it's just a series of binary classifications.

#### Disadvantages of OvR:

- **Imbalanced Training**: Since each binary classifier is trained against "the rest," this often creates imbalanced datasets (one class is much smaller compared to the others).

- **Confusion in close classes**: If two classes are very similar, OvR might struggle because the model isn’t directly comparing them to each other.

---

### What is One-vs-One (OvO)?

#### How it works:

- **One-vs-One** is a strategy where a binary classifier is trained for **every possible pair of classes**. For **n classes**, we build **n(n-1)/2** classifiers.

For the same example with 3 classes (A, B, and C):

- One classifier will predict **"Class A vs Class B"**.

- Another will predict **"Class A vs Class C"**.

- Another will predict **"Class B vs Class C"**.

#### Predictions:

- During prediction, each classifier votes for one of the two classes. The class that receives the **most votes** is chosen as the final prediction.

#### Advantages of OvO:

- **Better comparisons**: Since each classifier is trained only on two classes, the model can focus on distinguishing similar classes more effectively.

- **Balanced data**: Each binary classifier has a balanced dataset, as it’s only concerned with two classes at a time.

#### Disadvantages of OvO:

- **Scalability**: For a large number of classes, the number of classifiers grows significantly, which increases computational cost and complexity.

- **Prediction Time**: At prediction time, all classifiers have to run, which can be slower compared to OvR.

---

### OvO vs. OvR: Key Differences

| Feature | One-vs-Rest (OvR) | One-vs-One (OvO) |

|----------------------------|-----------------------------------------|---------------------------------------|

| **Number of Classifiers** | n (one for each class) | n(n-1)/2 (one for each pair of classes) |

| **Training Dataset Size** | Each classifier trained on full dataset | Each classifier trained on only two classes |

| **Prediction Approach** | Class with the highest confidence score | Class with the most votes |

| **Scalability** | More scalable for large numbers of classes | Can become computationally expensive with many classes |

| **Handling Similar Classes**| May struggle with very similar classes | Better at distinguishing between similar classes |

| **Training Time** | Faster due to fewer classifiers | Slower due to many classifiers |

| **Prediction Time** | Faster (just n classifiers) | Slower (all n(n-1)/2 classifiers run) |

---

### When to Use OvR vs. OvO?

#### Use **One-vs-Rest (OvR)** when:

- You have a **large number of classes** and need a simpler, faster solution.

- The problem doesn’t have many closely related classes.

- You’re working with classifiers that can handle imbalanced data well, such as logistic regression or decision trees.

#### Use **One-vs-One (OvO)** when:

- You have a **smaller number of classes** (e.g., less than 10), and computation is not a major concern.

- Classes are **closely related**, and you need a method that can more effectively distinguish between similar classes (e.g., for image or text classification tasks).

- You’re using models like **SVMs**, where OvO tends to work better due to the nature of SVM optimization.

---

### Conclusion

Both **OvO** and **OvR** are effective strategies for solving multiclass classification problems using binary classifiers. The choice between them depends largely on the size of the dataset, the number of classes, the nature of the classes, and the computational resources available. 

- For **larger datasets with many classes**, OvR is typically more efficient and easier to scale.

- For **smaller datasets with closely related classes**, OvO provides better class comparisons and often better performance.

Understanding the strengths and limitations of each method helps ensure you make the right choice for your specific classification problem.

Precision vs Recall: What to Use and When

Interactive Guide: Precision vs Recall vs F1 Score

Precision and Recall are essential metrics for evaluating classification models. This interactive guide explains how they work and lets you experiment with them.

---

Confusion Matrix Overview

Most classification metrics are derived from a confusion matrix.

True Positive (TP)
Correct positive prediction

False Positive (FP)
Incorrect positive prediction

False Negative (FN)
Missed positive case

True Negative (TN)
Correct negative prediction

---

1. Precision

Precision measures the proportion of correctly predicted positive instances out of all predicted positives.

Precision = True Positives / (True Positives + False Positives)

🔍 When to Use Precision

• False positives are costly
• You want high confidence in positive predictions
• Examples: spam filtering, medical diagnoses

$ evaluate_model --metric precision

True Positives: 80
False Positives: 20

Precision = 80 / (80 + 20)
Precision = 0.80

💡 High precision means very few incorrect positive predictions.

---

2. Recall

Recall measures how many actual positive cases were successfully identified.

Recall = True Positives / (True Positives + False Negatives)

🔍 When to Use Recall

• False negatives are costly
• You want to detect as many positives as possible
• Examples: cancer detection, fraud detection

$ evaluate_model --metric recall

True Positives: 80
False Negatives: 10

Recall = 80 / (80 + 10)
Recall = 0.89

💡 High recall means very few real positives are missed.

---

3. F1 Score (Balance between Precision and Recall)

F1 Score is the harmonic mean of precision and recall. It balances both metrics into one number.

F1 Score = 2 * (Precision * Recall) / (Precision + Recall)

🔍 Why F1 Score Matters

• Useful when classes are imbalanced
• Balances false positives and false negatives
• Common in machine learning competitions

$ evaluate_model --metric f1

Precision: 0.80
Recall: 0.89

F1 Score = 2 * (0.80 * 0.89) / (0.80 + 0.89)
F1 Score = 0.84

💡 F1 Score gives a balanced evaluation when both precision and recall matter.

---

Interactive Metric Simulator

Adjust the sliders below to simulate model predictions and see how the metrics change.

True Positives:

False Positives:

False Negatives:

Live Metrics

Precision:

Recall:

F1 Score:

💡 Try increasing False Positives to see precision drop. 💡 Increase False Negatives to see recall drop.

---

Summary

• Precision measures prediction accuracy for positives.
• Recall measures how many real positives are detected.
• F1 Score balances both metrics.

💡 The best metric depends on the real-world cost of false positives and false negatives.