Chi-Square Test for Categorical Data

Chi-Square Test, Degrees of Freedom, Normalization vs Standardization & Tobit Regression Explained

Statistics forms the mathematical backbone of modern data science, machine learning, econometrics, and research analysis. Among the most important statistical concepts are the Chi-Square Test, Degrees of Freedom, Normalization, Standardization, and advanced regression methods like the Tobit Model.

Understanding these concepts allows analysts to evaluate relationships between variables, preprocess datasets correctly, test hypotheses, and build more reliable predictive systems.

Key Learning Goals:

• Understand contingency tables
• Learn Chi-Square hypothesis testing
• Calculate expected frequencies
• Interpret p-values and significance
• Understand normalization and standardization
• Learn Wald and Likelihood Ratio tests in Tobit regression
• Implement everything in Python

---

Table of Contents

• 1. Introduction to Chi-Square Testing
• 2. Contingency Tables
• 3. Expected Frequencies
• 4. Chi-Square Statistic Formula
• 5. Degrees of Freedom
• 6. Hypothesis Testing
• 7. Interpreting Results
• 8. Goodness-of-Fit Test
• 9. Test of Independence
• 10. Normalization
• 11. Standardization
• 12. Normalization vs Standardization
• 13. Tobit Regression
• 14. Wald Chi-Square Test
• 15. Likelihood Ratio Test
• 16. Python Examples
• 17. CLI Outputs
• 18. Interactive Learning
• 19. Final Conclusion

---

1. Introduction to Chi-Square Testing

The Chi-Square test is one of the most widely used statistical methods for categorical data analysis.

It helps answer questions like:

• Do boys and girls prefer different colors?
• Is customer preference related to age group?
• Is there an association between smoking and disease?
• Are observed outcomes significantly different from expected outcomes?

The Chi-Square test compares:

• Observed frequencies
• Expected frequencies

If the difference between them is large enough, the relationship is considered statistically significant.

---

2. Contingency Tables

A contingency table organizes categorical data into rows and columns.

	Green	Pink	Blue	Total
Boys	O1	O2	O3	B
Girls	O4	O5	O6	G
Total	T1	T2	T3	N

This table allows researchers to evaluate whether gender and color preference are associated.

---

3. Expected Frequencies

Expected frequencies represent the values we would expect if there were no relationship between variables.

\[ E_{ij} = \frac{(Row\ Total_i)(Column\ Total_j)}{Grand\ Total} \]

Example

\[ E_{Boys,Green} = \frac{B \times T1}{N} \]

If observed values differ significantly from expected values, the variables may be associated.

---

4. Chi-Square Statistic Formula

\[ \chi^2 = \sum \frac{(O_{ij} - E_{ij})^2}{E_{ij}} \]

Where:

• \(O_{ij}\) = Observed frequency
• \(E_{ij}\) = Expected frequency

The Chi-Square statistic measures deviation between observed and expected frequencies.

Larger Chi-Square values indicate stronger evidence against the null hypothesis.

---

5. Degrees of Freedom

Degrees of freedom determine which Chi-Square distribution should be used.

Formula for Independence Test

\[ df = (Rows - 1)(Columns - 1) \]

Example

\[ df = (2-1)(3-1)=2 \]

Goodness-of-Fit Formula

\[ df = Categories - 1 \]

---

6. Hypothesis Testing

Null Hypothesis

Variables are independent.

Alternative Hypothesis

Variables are associated.

Decision rules:

• p-value < 0.05 → reject null hypothesis
• p-value > 0.05 → fail to reject null hypothesis

---

7. Interpreting Results

Condition	Interpretation
Large Chi-Square	Strong evidence of association
Small Chi-Square	Weak evidence of association
p-value < 0.05	Statistically significant
p-value > 0.05	Not statistically significant

---

8. Goodness-of-Fit Test

This test evaluates whether observed frequencies match a theoretical distribution.

Example

Suppose a die is rolled 60 times. Expected frequency:

\[ E = \frac{60}{6}=10 \]

Chi-Square determines whether deviations from 10 are due to chance.

---

9. Test of Independence

This test evaluates whether two categorical variables are related.

Examples:

• Gender vs color preference
• Education vs income level
• Smoking vs lung disease

---

10. Normalization

Normalization rescales values into a fixed range, usually:

\[ [0,1] \]

Formula

\[ X' = \frac{X - X_{min}}{X_{max}-X_{min}} \]

Normalization is useful when:

• Features have different scales
• Neural networks are used
• Distance-based models are applied

---

11. Standardization

Standardization transforms data to:

• Mean = 0
• Standard deviation = 1

Formula

\[ Z = \frac{X-\mu}{\sigma} \]

Where:

• \(\mu\) = mean
• \(\sigma\) = standard deviation

---

12. Normalization vs Standardization

Feature	Normalization	Standardization
Range	0 to 1	No fixed range
Mean	Not fixed	0
Standard Deviation	Not fixed	1
Outlier Sensitivity	High	Moderate
Best Used For	Neural Networks	Regression Models

---

13. Tobit Regression

Tobit regression is designed for censored dependent variables.

Examples:

• Income censored at zero
• Survey scores with upper limits
• Measurements below detection thresholds

\[ Y_i^* = X_i \beta + \epsilon_i \]

Observed:

\[ Y_i = \begin{cases} Y_i^* & \text{if } Y_i^* > 0 \\ 0 & \text{otherwise} \end{cases} \]

---

14. Wald Chi-Square Test

The Wald test evaluates whether coefficients differ significantly from zero.

\[ \chi^2 = \left(\frac{\beta}{SE}\right)^2 \]

Where:

• \(\beta\) = coefficient estimate
• \(SE\) = standard error

Large Wald Chi-Square values imply statistically significant predictors.

---

15. Likelihood Ratio Test

Likelihood Ratio Test compares:

• Full model
• Restricted model

\[ \chi^2 = -2(LL_{restricted}-LL_{full}) \]

Where:

• \(LL\) = Log-likelihood

If the statistic is large, the full model significantly improves prediction.

---

16. Python Examples

Chi-Square Test Example

import scipy.stats as stats

observed = [[20, 15, 10],
            [10, 25, 20]]

chi2, p, dof, expected = stats.chi2_contingency(observed)

print("Chi-Square:", chi2)
print("p-value:", p)
print("Degrees of Freedom:", dof)
print("Expected Frequencies:")
print(expected)

Tobit Regression Example

import numpy as np
import scipy.stats as stats
from tobit import TobitModel

model = TobitModel(y, X, left=0)
results = model.fit()

betas = results.params_
se = results.bse_

wald_chi2 = (betas / se) ** 2

p_values = 1 - stats.chi2.cdf(wald_chi2, df=1)

for i, (beta, chi2, p) in enumerate(zip(betas, wald_chi2, p_values)):
    print(f"Variable {i}: Chi-Square = {chi2:.4f}, p-value = {p:.4f}")

---

17. CLI Outputs

$ python chi_square_test.py

Chi-Square: 8.512
p-value: 0.014
Degrees of Freedom: 2

Result:
Reject the null hypothesis.
Variables are significantly associated.

$ python tobit_model.py

Variable 1:
Chi-Square = 14.8821
p-value = 0.0001

Variable 2:
Chi-Square = 0.9912
p-value = 0.3192

---

18. Interactive Learning Section

Expected frequencies represent what should occur if variables are independent. Comparing observed values to expected values helps determine whether relationships are statistically meaningful.

Degrees of freedom determine the correct Chi-Square distribution shape. Incorrect degrees of freedom lead to invalid statistical conclusions.

Normalization works best when features have bounded ranges or when neural networks and distance-based algorithms are used.

Ordinary regression fails with censored datasets. Tobit regression correctly handles variables limited by upper or lower thresholds.

---

19. Final Conclusion

Chi-Square testing is fundamental for categorical data analysis and hypothesis testing. It helps determine whether observed relationships are statistically significant or simply due to random variation.

Degrees of freedom ensure proper statistical interpretation, while normalization and standardization prepare datasets for machine learning algorithms.

Advanced models like Tobit regression extend statistical analysis into censored datasets, enabling researchers to analyze limited or constrained outcomes effectively.

Final Summary:

• Chi-Square compares observed vs expected frequencies.
• Degrees of freedom determine the statistical distribution.
• Normalization rescales values between 0 and 1.
• Standardization creates mean 0 and variance 1.
• Tobit regression handles censored dependent variables.
• Wald and Likelihood Ratio tests evaluate model significance.