Chi-Square Test, Degrees of Freedom, Normalization vs Standardization & Tobit Regression Explained
Statistics forms the mathematical backbone of modern data science, machine learning, econometrics, and research analysis. Among the most important statistical concepts are the Chi-Square Test, Degrees of Freedom, Normalization, Standardization, and advanced regression methods like the Tobit Model.
Understanding these concepts allows analysts to evaluate relationships between variables, preprocess datasets correctly, test hypotheses, and build more reliable predictive systems.
- Understand contingency tables
- Learn Chi-Square hypothesis testing
- Calculate expected frequencies
- Interpret p-values and significance
- Understand normalization and standardization
- Learn Wald and Likelihood Ratio tests in Tobit regression
- Implement everything in Python
Table of Contents
- 1. Introduction to Chi-Square Testing
- 2. Contingency Tables
- 3. Expected Frequencies
- 4. Chi-Square Statistic Formula
- 5. Degrees of Freedom
- 6. Hypothesis Testing
- 7. Interpreting Results
- 8. Goodness-of-Fit Test
- 9. Test of Independence
- 10. Normalization
- 11. Standardization
- 12. Normalization vs Standardization
- 13. Tobit Regression
- 14. Wald Chi-Square Test
- 15. Likelihood Ratio Test
- 16. Python Examples
- 17. CLI Outputs
- 18. Interactive Learning
- 19. Final Conclusion
1. Introduction to Chi-Square Testing
The Chi-Square test is one of the most widely used statistical methods for categorical data analysis.
It helps answer questions like:
- Do boys and girls prefer different colors?
- Is customer preference related to age group?
- Is there an association between smoking and disease?
- Are observed outcomes significantly different from expected outcomes?
The Chi-Square test compares:
- Observed frequencies
- Expected frequencies
If the difference between them is large enough, the relationship is considered statistically significant.
2. Contingency Tables
A contingency table organizes categorical data into rows and columns.
| Green | Pink | Blue | Total | |
|---|---|---|---|---|
| Boys | O1 | O2 | O3 | B |
| Girls | O4 | O5 | O6 | G |
| Total | T1 | T2 | T3 | N |
This table allows researchers to evaluate whether gender and color preference are associated.
3. Expected Frequencies
Expected frequencies represent the values we would expect if there were no relationship between variables.
Example
If observed values differ significantly from expected values, the variables may be associated.
4. Chi-Square Statistic Formula
Where:
- \(O_{ij}\) = Observed frequency
- \(E_{ij}\) = Expected frequency
The Chi-Square statistic measures deviation between observed and expected frequencies.
5. Degrees of Freedom
Degrees of freedom determine which Chi-Square distribution should be used.
Formula for Independence Test
Example
Goodness-of-Fit Formula
6. Hypothesis Testing
Null Hypothesis
Variables are independent.
Alternative Hypothesis
Variables are associated.
Decision rules:
- p-value < 0.05 → reject null hypothesis
- p-value > 0.05 → fail to reject null hypothesis
7. Interpreting Results
| Condition | Interpretation |
|---|---|
| Large Chi-Square | Strong evidence of association |
| Small Chi-Square | Weak evidence of association |
| p-value < 0.05 | Statistically significant |
| p-value > 0.05 | Not statistically significant |
8. Goodness-of-Fit Test
This test evaluates whether observed frequencies match a theoretical distribution.
Example
Suppose a die is rolled 60 times. Expected frequency:
Chi-Square determines whether deviations from 10 are due to chance.
9. Test of Independence
This test evaluates whether two categorical variables are related.
Examples:
- Gender vs color preference
- Education vs income level
- Smoking vs lung disease
10. Normalization
Normalization rescales values into a fixed range, usually:
Formula
Normalization is useful when:
- Features have different scales
- Neural networks are used
- Distance-based models are applied
11. Standardization
Standardization transforms data to:
- Mean = 0
- Standard deviation = 1
Formula
Where:
- \(\mu\) = mean
- \(\sigma\) = standard deviation
12. Normalization vs Standardization
| Feature | Normalization | Standardization |
|---|---|---|
| Range | 0 to 1 | No fixed range |
| Mean | Not fixed | 0 |
| Standard Deviation | Not fixed | 1 |
| Outlier Sensitivity | High | Moderate |
| Best Used For | Neural Networks | Regression Models |
13. Tobit Regression
Tobit regression is designed for censored dependent variables.
Examples:
- Income censored at zero
- Survey scores with upper limits
- Measurements below detection thresholds
Observed:
14. Wald Chi-Square Test
The Wald test evaluates whether coefficients differ significantly from zero.
Where:
- \(\beta\) = coefficient estimate
- \(SE\) = standard error
15. Likelihood Ratio Test
Likelihood Ratio Test compares:
- Full model
- Restricted model
Where:
- \(LL\) = Log-likelihood
If the statistic is large, the full model significantly improves prediction.
16. Python Examples
Chi-Square Test Example
import scipy.stats as stats
observed = [[20, 15, 10],
[10, 25, 20]]
chi2, p, dof, expected = stats.chi2_contingency(observed)
print("Chi-Square:", chi2)
print("p-value:", p)
print("Degrees of Freedom:", dof)
print("Expected Frequencies:")
print(expected)
Tobit Regression Example
import numpy as np
import scipy.stats as stats
from tobit import TobitModel
model = TobitModel(y, X, left=0)
results = model.fit()
betas = results.params_
se = results.bse_
wald_chi2 = (betas / se) ** 2
p_values = 1 - stats.chi2.cdf(wald_chi2, df=1)
for i, (beta, chi2, p) in enumerate(zip(betas, wald_chi2, p_values)):
print(f"Variable {i}: Chi-Square = {chi2:.4f}, p-value = {p:.4f}")
17. CLI Outputs
$ python chi_square_test.py
Chi-Square: 8.512
p-value: 0.014
Degrees of Freedom: 2
Result:
Reject the null hypothesis.
Variables are significantly associated.
$ python tobit_model.py
Variable 1:
Chi-Square = 14.8821
p-value = 0.0001
Variable 2:
Chi-Square = 0.9912
p-value = 0.3192
18. Interactive Learning Section
Expected frequencies represent what should occur if variables are independent. Comparing observed values to expected values helps determine whether relationships are statistically meaningful.
Degrees of freedom determine the correct Chi-Square distribution shape. Incorrect degrees of freedom lead to invalid statistical conclusions.
Normalization works best when features have bounded ranges or when neural networks and distance-based algorithms are used.
Ordinary regression fails with censored datasets. Tobit regression correctly handles variables limited by upper or lower thresholds.
19. Final Conclusion
Chi-Square testing is fundamental for categorical data analysis and hypothesis testing. It helps determine whether observed relationships are statistically significant or simply due to random variation.
Degrees of freedom ensure proper statistical interpretation, while normalization and standardization prepare datasets for machine learning algorithms.
Advanced models like Tobit regression extend statistical analysis into censored datasets, enabling researchers to analyze limited or constrained outcomes effectively.
- Chi-Square compares observed vs expected frequencies.
- Degrees of freedom determine the statistical distribution.
- Normalization rescales values between 0 and 1.
- Standardization creates mean 0 and variance 1.
- Tobit regression handles censored dependent variables.
- Wald and Likelihood Ratio tests evaluate model significance.
