Comparing Age Distributions: 2000 Census vs. 2010 Sample and Chi-Square Test Analysis

Chi-Square Test, Z Scores, Standard Deviation & Data Visualization Explained

Statistics forms the foundation of modern data science, machine learning, economics, demographics, and predictive analytics. Whether analyzing census data, customer behavior, medical trials, or business trends, understanding statistical concepts helps transform raw numbers into meaningful insights.

In this comprehensive guide, we will explore:

• Chi-Square Test
• Observed vs Expected Values
• Z Scores
• Standard Deviation
• Demographic Analysis
• Python Data Visualization
• Histograms and Boxplots
• Statistical Interpretation

Key Learning Goal:
By the end of this tutorial, you will understand how to compare distributions statistically, measure deviations from the mean, and visualize demographic patterns using Python.

---

Table of Contents

• 1. Introduction to Statistical Distribution
• 2. What is Chi-Square Test?
• 3. Observed vs Expected Values
• 4. Chi-Square Calculation
• 5. Interpreting Chi-Square Results
• 6. Understanding Z Scores
• 7. Standard Deviation Explained
• 8. Normal Distribution
• 9. Variance and Dispersion
• 10. Python Visualization
• 11. Histograms
• 12. Boxplots
• 13. Advanced Statistical Concepts
• 14. CLI Examples
• 15. Final Conclusion

---

1. Introduction to Statistical Distribution

Statistics helps us understand patterns within data. One of the most important questions analysts ask is:

Does the observed data behave the way we expected?

For example:

• Did the population age distribution change?
• Are customer preferences random or structured?
• Are exam scores unusually high?
• Is a sample representative of the population?

Statistical tests help answer these questions objectively.

---

2. What is Chi-Square Test?

The Chi-Square test is one of the most widely used statistical tests for categorical data.

It measures how different observed frequencies are from expected frequencies.

Main Purpose

• Compare observed vs expected distributions
• Test relationships between categorical variables
• Identify statistically significant differences

\[ \chi^2 = \sum \frac{(O_i - E_i)^2}{E_i} \]

Where:

• \(O_i\) = Observed frequency
• \(E_i\) = Expected frequency

---

3. Observed vs Expected Values

Observed Value

Observed values are actual measurements collected from data.

Age Group	Observed Count
Age 18	121
Age 18-35	288
Age >35	91

Expected Value

Expected values are theoretical counts assuming the historical proportions remain unchanged.

\[ E_i = p_i \times n \]

Where:

• \(p_i\) = historical proportion
• \(n\) = sample size

Age Group	Expected Count
Age 18	100
Age 18-35	150
Age >35	250

---

4. Chi-Square Calculation

Now we calculate the difference between observed and expected values.

Step 1: Age 18

\[ \frac{(121 - 100)^2}{100} = \frac{441}{100} = 4.41 \]

Step 2: Age 18-35

\[ \frac{(288 - 150)^2}{150} = \frac{19044}{150} \approx 127.4 \]

Step 3: Age >35

\[ \frac{(91 - 250)^2}{250} = \frac{25281}{250} \approx 101.1 \]

Final Chi-Square Statistic

\[ \chi^2 = 4.41 + 127.4 + 101.1 = 232.91 \]

---

5. Interpreting Chi-Square Results

A large Chi-Square statistic means observed data differs significantly from expected data.

Degrees of Freedom

\[ df = k - 1 \]

Where:

• \(k\) = number of categories

For 3 categories:

\[ df = 3 - 1 = 2 \]

Interpretation

• Large Chi-Square → Significant difference
• Small Chi-Square → Differences due to randomness

The 2010 sample strongly differs from the 2000 census distribution, suggesting a major demographic shift toward younger individuals.

---

6. Understanding Z Scores

Z Scores measure how far a value lies from the mean in terms of standard deviations.

\[ z = \frac{x - \mu}{\sigma} \]

Where:

• \(x\) = observed value
• \(\mu\) = mean
• \(\sigma\) = standard deviation

Interpretation

Z Score	Meaning
0	Exactly at mean
+1	1 standard deviation above mean
-1	1 standard deviation below mean
+2	Far above average
-2	Far below average

---

7. Standard Deviation Explained

Standard deviation measures how spread out data points are.

Formula

\[ \sigma = \sqrt{ \frac{ \sum (x_i - \mu)^2 }{n} } \]

Small standard deviation:

• Data clustered near mean

Large standard deviation:

• Data widely spread

Simple Example

Dataset A:

48, 49, 50, 51, 52

Very low standard deviation.

Dataset B:

10, 30, 50, 70, 90

Very high standard deviation.

---

8. Normal Distribution

Many real-world variables approximately follow a normal distribution.

\[ f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} \]

Characteristics:

• Bell-shaped curve
• Mean = Median = Mode
• Symmetrical distribution

68-95-99 Rule

• 68% within 1 standard deviation
• 95% within 2 standard deviations
• 99.7% within 3 standard deviations

---

9. Variance and Dispersion

Variance measures average squared deviation from the mean.

\[ \sigma^2 = \frac{ \sum (x_i - \mu)^2 }{n} \]

Standard deviation is simply the square root of variance.

---

10. Python Visualization

Import Required Libraries

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Create Sample Dataset

data = {
    'age': [25,30,22,35,40,28,50,19,60,45,32,37,29,23,41],
    'gender': [
        'Male','Female','Male','Female',
        'Male','Female','Male','Female',
        'Male','Female','Male','Female',
        'Male','Female','Male'
    ]
}

df = pd.DataFrame(data)

---

11. Histogram Visualization

Histograms show frequency distribution.

plt.figure(figsize=(10,6))

sns.histplot(
    data=df,
    x='age',
    hue='gender',
    kde=True,
    bins=10,
    alpha=0.6
)

plt.xlabel('Age')
plt.ylabel('Count')
plt.title('Age Distribution by Gender')

plt.show()

Explanation

• hue separates gender categories
• kde=True adds density curve
• bins=10 divides age into intervals
• alpha controls transparency

---

12. Boxplot Visualization

Boxplots summarize distributions compactly.

plt.figure(figsize=(8,6))

sns.boxplot(
    x='gender',
    y='age',
    data=df
)

plt.xlabel('Gender')
plt.ylabel('Age')
plt.title('Age Distribution by Gender')

plt.show()

Boxplot Components

• Median
• Quartiles
• Interquartile range
• Outliers

---

13. Advanced Statistical Concepts

P-Value

P-value measures probability of observing results assuming the null hypothesis is true.

\[ p < 0.05 \]

Usually considered statistically significant.

Hypothesis Testing

• Null Hypothesis (\(H_0\))
• Alternative Hypothesis (\(H_1\))

Sampling Error

Differences between sample statistics and population parameters.

Central Limit Theorem

\[ \bar{X} \sim N\left( \mu, \frac{\sigma}{\sqrt{n}} \right) \]

Large samples approximate normal distributions.

---

14. CLI Output Examples

Chi-Square Test Output

$ python chi_square_test.py

Observed:
[121, 288, 91]

Expected:
[100, 150, 250]

Chi-Square Statistic:
232.91

Degrees of Freedom:
2

P-value:
< 0.0001

Conclusion:
Reject the null hypothesis.
The distributions are significantly different.

Z Score Calculation Output

$ python zscore.py

Mean Age: 35
Standard Deviation: 8

Age Selected: 51

Z Score:
2.00

Interpretation:
The value is 2 standard deviations above the mean.

---

Interactive Learning Section

Standard deviation measures data variability. It helps determine whether observations are tightly grouped or highly dispersed.

Chi-Square compares category frequencies rather than continuous numerical values, making it ideal for demographic and survey analysis.

A large absolute Z score indicates an observation far from the average, potentially identifying unusual or rare events.

---

15. Final Conclusion

Statistics provides the mathematical language for understanding uncertainty, variation, and patterns in data.

The Chi-Square test helps determine whether observed categorical distributions differ significantly from expectations. Z Scores and standard deviation help measure how far observations deviate from the average.

Together, these concepts form the backbone of:

• Machine Learning
• Data Science
• Forecasting
• Business Analytics
• Demographic Research
• Economics
• AI Systems

Final Learning Summary:

• Chi-Square compares observed and expected frequencies.
• Z Scores measure distance from the mean.
• Standard deviation measures spread.
• Histograms visualize frequency distributions.
• Boxplots identify medians and outliers.
• Python visualization improves statistical interpretation.
• Statistical significance helps validate conclusions.