## **Identifying and Handling Column Data Types in the Titanic Dataset**
When working with datasets like the Titanic dataset, it's important to distinguish between categorical and numerical data types because they often require different preprocessing steps. Below is a step-by-step explanation of how to identify and handle these different types of columns using Pandas.
### **Step 1: Check Data Types**
The first step is to identify the data types of each column in the dataset. This helps in determining which columns are categorical and which are numerical.
import pandas as pd
# Load the Titanic dataset
df = pd.read_csv("titanic.csv")
# Check the data types of each column
print(df.dtypes)
### **Step 2: Identify Categorical Columns**
Categorical columns typically store data as strings (Pandas `object` type). These columns can be identified and isolated for specific operations like encoding.
# Identify categorical columns by selecting those with 'object' data type
categorical_cols = df.select_dtypes(include=['object']).columns
print("Categorical Columns:", categorical_cols)
### **Step 3: Convert Categorical Columns to 'Category' Data Type**
Converting categorical columns from `object` to `category` can be beneficial. It reduces memory usage and can make operations on these columns faster.
# Convert categorical columns to 'category' data type
df[categorical_cols] = df[categorical_cols].astype('category')
# Verify the change
print(df.dtypes)
### **Step 4: Identify Numerical Columns**
Numerical columns, which store integer or float data, can also be isolated for specific numerical operations like normalization or scaling.
# Identify numerical columns
numerical_cols = df.select_dtypes(include=['int64', 'float64']).columns
print("Numerical Columns:", numerical_cols)
### **Why Is This Important?**
- **Memory Efficiency**: Converting to the `category` data type reduces the memory footprint of the DataFrame, which is especially useful with large datasets.
- **Targeted Operations**: By knowing the types of data in each column, you can apply the appropriate preprocessing techniques. For example, categorical data may need encoding, while numerical data might require scaling or normalization.
### **Using `describe()` on Categorical Columns**
You can also use the `describe()` method to get a summary of categorical columns. This method provides useful statistics like the number of unique values, the most frequent value, and its frequency.
# Use describe() on the entire DataFrame to include both categorical and numerical columns
description = df.describe(include='all')
print(description)
# Alternatively, describe only the categorical columns
categorical_description = df[categorical_cols].describe()
print(categorical_description)
### **Example Output**
Here’s an example of what the output might look like after running `describe()` on categorical columns:
Name Sex Cabin Embarked
count 891 891 204 889
unique 891 2 147 3
top Braund, Mr. Owen Harris male B96 B98 S
freq 1 577 4 644
### **Explanation of the Output:**
- **count**: Number of non-null entries in each column.
- **unique**: Number of unique categories within the column.
- **top**: The most common category (mode) in the column.
- **freq**: Frequency (count) of the top category.
### **Conclusion**
This approach of checking and converting data types is crucial in data preprocessing. It allows for more efficient data manipulation and prepares the data for machine learning algorithms, which often require numerical input. Understanding and managing categorical data types properly can also lead to better model performance and insights.
