Why You Shouldn't Use Standard Scaler on Categorical Data

StandardScaler, Categorical Data, Supervised vs Semi-Supervised Learning & Custom Scalar Product in NumPy

Data preprocessing is one of the most important stages in machine learning and data science. A powerful machine learning model can completely fail if the input data is poorly prepared. Many beginners focus heavily on algorithms while ignoring preprocessing, encoding, scaling, normalization, and optimization.

This guide explains:

• Why StandardScaler works only for numerical data
• Why categorical data should not be scaled
• How encoding works
• The difference between supervised and semi-supervised learning
• How to optimize custom scalar products using NumPy
• Vectorized computation strategies
• Mathematical foundations behind scaling and vectorization

Key Learning Goal:
Understanding preprocessing is often more important than choosing complex machine learning algorithms.

---

Table of Contents

• 1. What is StandardScaler?
• 2. Mathematical Foundation of Scaling
• 3. Understanding Categorical Data
• 4. Why Scaling Categorical Data is Wrong
• 5. Encoding Techniques
• 6. One-Hot Encoding
• 7. Label Encoding
• 8. Target Encoding
• 9. Numerical vs Categorical Features
• 10. Supervised Learning
• 11. Semi-Supervised Learning
• 12. Supervised vs Semi-Supervised
• 13. Custom Scalar Product
• 14. NumPy Vectorization
• 15. Universal Functions
• 16. Optimization Strategies
• 17. Python Code Examples
• 18. CLI Output Examples
• 19. Common Mistakes
• 20. Final Conclusion

---

1. What is StandardScaler?

StandardScaler is one of the most commonly used preprocessing tools in machine learning. It standardizes numerical data by transforming features into a common scale.

Its purpose is to:

• Center data around zero
• Normalize variance
• Improve model convergence
• Prevent features with large magnitudes from dominating

After scaling:

• Mean becomes 0
• Standard deviation becomes 1

\[ z = \frac{x - \mu}{\sigma} \]

Where:

• \(x\) = original value
• \(\mu\) = mean
• \(\sigma\) = standard deviation
• \(z\) = standardized value

---

2. Mathematical Foundation of Scaling

Mean Formula

\[ \mu = \frac{1}{n}\sum_{i=1}^{n}x_i \]

Variance Formula

\[ \sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \mu)^2 \]

Standard Deviation

\[ \sigma = \sqrt{\sigma^2} \]

Standardization transforms data distributions into comparable scales.

Scaling is critical for algorithms using distance calculations like KNN, SVM, and Logistic Regression.

---

3. Understanding Categorical Data

Categorical data represents labels or groups rather than measurable quantities.

Examples

• Colors
• Cities
• Animal types
• Product categories
• Gender labels

Types of Categorical Data

Type	Description	Example
Nominal	No order	Red, Blue, Green
Ordinal	Meaningful order	Low, Medium, High

---

4. Why Scaling Categorical Data is Wrong

The biggest mistake beginners make is scaling encoded categories.

Example:

City	Encoded
New York	1
London	2
Paris	3

These numbers are labels, not mathematical quantities.

Scaling them creates false relationships.

\[ z = \frac{x - \mu}{\sigma} \]

After scaling:

• New York may become -1.2
• London may become 0.3
• Paris may become 1.1

This incorrectly implies mathematical distance between cities.

Encoded numbers are identifiers, not measurable quantities.

---

5. Encoding Techniques

Machine learning algorithms require numerical input. Therefore categorical values must be encoded.

However, encoding is not scaling.

---

6. One-Hot Encoding

One-hot encoding creates binary columns.

City	City_London	City_Paris	City_NewYork
London	1	0	0
Paris	0	1	0

Advantages

• No false ordering
• Safe for nominal data
• Widely used

Disadvantages

• High dimensionality
• Sparse matrices

---

7. Label Encoding

Label encoding assigns integers.

Category	Encoded Value
Low	1
Medium	2
High	3

This works for ordinal data because ordering exists.

It becomes dangerous for nominal categories.

---

8. Target Encoding

Target encoding replaces categories using target statistics.

\[ Encoded(Category_i) = Mean(Target_i) \]

Useful for:

• High-cardinality features
• Large datasets
• Competition datasets

But it can cause data leakage if improperly implemented.

---

9. Numerical vs Categorical Features

Feature	Type	Scale?
Age	Numerical	Yes
Salary	Numerical	Yes
Temperature	Numerical	Yes
City	Categorical	No
Gender	Categorical	No

---

10. Supervised Learning

Supervised learning uses labeled data.

\[ (X, y) \]

Where:

• \(X\) = features
• \(y\) = target labels

Examples

• Email spam detection
• House price prediction
• Image classification
• Credit risk analysis

Popular Algorithms

• Linear Regression
• Logistic Regression
• Decision Trees
• Random Forest
• SVM
• Neural Networks

---

11. Semi-Supervised Learning

Semi-supervised learning combines:

• Small labeled datasets
• Large unlabeled datasets

This approach reduces labeling costs.

\[ D = D_L \cup D_U \]

Where:

• \(D_L\) = labeled data
• \(D_U\) = unlabeled data

Applications

• Medical imaging
• Speech recognition
• Fraud detection
• Web classification

---

12. Supervised vs Semi-Supervised Learning

Aspect	Supervised	Semi-Supervised
Labels Required	All data	Partial data
Cost	High	Lower
Accuracy	High with enough labels	Good with limited labels
Complexity	Moderate	Higher

---

13. Custom Scalar Product

The traditional dot product is:

\[ a \cdot b = \sum_{i=1}^{n} a_i b_i \]

Sometimes custom multiplication and addition are required.

\[ a \otimes b = my\_add(my\_mult(a_1,b_1), my\_mult(a_2,b_2)) \]

This is common in:

• Cryptography
• Error correction
• Finite field algebra
• Signal processing

---

14. NumPy Vectorization

Loops are slow in Python.

NumPy vectorization performs operations in optimized C-level code.

Traditional Loop

result = 0

for i in range(len(a)):
    result += a[i] * b[i]

Vectorized Version

result = np.dot(a, b)

Vectorized operations are dramatically faster than Python loops.

---

15. Universal Functions (ufuncs)

NumPy ufuncs apply operations element-wise.

Example

import numpy as np

def my_mult(x, y):
    return x ^ y

ufunc_mult = np.frompyfunc(my_mult, 2, 1)

This converts Python functions into NumPy-compatible vectorized functions.

---

16. Optimization Strategies

1. Early Conversion

Convert strings into integers early.

2. Bulk Operations

Avoid element-wise Python loops.

3. Broadcasting

Leverage NumPy broadcasting rules.

\[ A_{m \times n} + B_{1 \times n} \]

Broadcasting expands dimensions automatically.

4. Reduction Operations

np.add.reduce(array)

---

17. Python Code Examples

StandardScaler Example

from sklearn.preprocessing import StandardScaler
import pandas as pd

data = pd.DataFrame({
    "Age":[22,25,30,35],
    "Salary":[25000,40000,60000,80000]
})

scaler = StandardScaler()

scaled = scaler.fit_transform(data)

print(scaled)

One-Hot Encoding Example

import pandas as pd

df = pd.DataFrame({
    "City":["London","Paris","New York"]
})

encoded = pd.get_dummies(df)

print(encoded)

Custom Scalar Product Example

import numpy as np

a = np.array([1,2,3])
b = np.array([4,5,6])

def my_mult(x,y):
    return x * y

def my_add(x,y):
    return x + y

products = np.vectorize(my_mult)(a,b)

result = np.add.reduce(products)

print(result)

---

18. CLI Output Examples

$ python scaler.py

Original Data:
[[22,25000],
 [25,40000],
 [30,60000]]

Scaled Data:
[[-1.12,-1.02],
 [-0.52,-0.32],
 [ 0.75, 0.85]]

$ python encoding.py

City_London  City_Paris  City_NewYork
1            0           0
0            1           0
0            0           1

$ python scalar_product.py

Custom Scalar Product Result:
32

---

Interactive Learning Section

Many algorithms rely on distances and gradients. Features with larger magnitudes dominate optimization if scaling is ignored.

It prevents false ordering relationships between categories and preserves category independence.

Python loops execute interpreted code repeatedly, while NumPy executes optimized compiled operations internally using vectorization.

---

19. Common Mistakes

• Scaling categorical variables
• Using label encoding for nominal data
• Ignoring feature scaling for distance-based algorithms
• Using loops instead of vectorization
• Performing string conversion repeatedly inside loops
• Ignoring data leakage during target encoding

Efficient preprocessing can improve models more than changing algorithms.

---

20. Final Conclusion

Understanding preprocessing is one of the most important skills in machine learning and data science.

StandardScaler is powerful for numerical features because numerical values possess meaningful distances and distributions. However, categorical variables represent labels, not measurable quantities, making scaling inappropriate.

Choosing the correct encoding strategy prevents distorted feature relationships and improves model reliability.

Similarly, efficient computation using NumPy requires understanding vectorization, broadcasting, and optimized array operations rather than relying on slow Python loops.

Whether building machine learning pipelines or scientific computing systems, mastering preprocessing and optimization fundamentals is essential for high-performance data science workflows.

Final Learning Summary:

• StandardScaler should only be used for numerical data.
• Categorical labels do not have mathematical meaning.
• One-hot encoding is safest for nominal categories.
• Semi-supervised learning uses both labeled and unlabeled data.
• NumPy vectorization dramatically improves performance.
• Broadcasting and ufuncs reduce computational overhead.
• Efficient preprocessing leads to better models.