Data Wrangling & Preprocessing

The Wrangling Workflow

Data wrangling follows a predictable pattern. Start by understanding what you have, then fix issues systematically.

Step 1: Inspect Your Data

Before fixing anything, understand what you're working with. These five commands tell you almost everything you need to know.

import pandas as pd

df = pd.read_csv('dataset.csv')

# Shape: how many rows and columns?
print(df.shape)              # (10000, 15)

# Column types and null counts
print(df.info())

# Statistical summary of numeric columns
print(df.describe())

# First few rows
print(df.head())

# Unique values per column (spot unexpected categories)
for col in df.select_dtypes(include='object'):
    print(f"{col}: {df[col].nunique()} unique — {df[col].unique()[:5]}")

Step 2: Handle Missing Values

Missing data is the most common issue. Your strategy depends on how much is missing and why it's missing.

# Visualize missing data patterns
missing_pct = (df.isnull().sum() / len(df) * 100).sort_values(ascending=False)
print(missing_pct[missing_pct > 0])

# Strategy: create a "was_missing" flag, then impute
df['income_missing'] = df['income'].isnull().astype(int)
df['income'].fillna(df['income'].median(), inplace=True)

# For categorical columns, fill with the mode
df['city'].fillna(df['city'].mode()[0], inplace=True)

# For time series, use forward fill
df['temperature'].ffill(inplace=True)

Step 3: Fix Data Types

Wrong types silently break calculations. A "price" column stored as a string can't be summed or averaged.

# String prices → numeric (remove $, commas)
df['price'] = df['price'].str.replace('[$,]', '', regex=True).astype(float)

# String dates → datetime
df['date'] = pd.to_datetime(df['date'], format='%Y-%m-%d')

# Categories with few unique values → category type (saves memory)
df['status'] = df['status'].astype('category')

# Boolean strings → actual booleans
df['is_active'] = df['is_active'].map({'yes': True, 'no': False, 'Yes': True, 'No': False})

Step 4: Remove Duplicates

Duplicates inflate your dataset and bias your model. Check for exact duplicates and near-duplicates (same entity, slightly different data).

# Exact duplicates
print(f"Duplicates: {df.duplicated().sum()}")
df.drop_duplicates(inplace=True)

# Duplicates on specific columns (e.g., same customer, same date)
df.drop_duplicates(subset=['customer_id', 'order_date'], keep='last', inplace=True)

# Near-duplicates: check for similar but not identical records
df[df.duplicated(subset=['name', 'email'], keep=False)].sort_values('name')

Step 5: Handle Outliers

Outliers are extreme values that can skew your analysis and mislead your model. Detect them with IQR or Z-scores, then decide: keep, cap, or remove.

import numpy as np

# Method 1: IQR (Interquartile Range)
Q1 = df['salary'].quantile(0.25)
Q3 = df['salary'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

outliers = df[(df['salary'] < lower) | (df['salary'] > upper)]
print(f"Outliers found: {len(outliers)}")

# Method 2: Z-score (assumes normal distribution)
from scipy import stats
z_scores = np.abs(stats.zscore(df['salary'].dropna()))
outlier_mask = z_scores > 3  # beyond 3 standard deviations

# Strategy: Cap (winsorize) instead of removing
df['salary'] = df['salary'].clip(lower=lower, upper=upper)

Step 6: Reshape Data

Sometimes data arrives in the wrong shape. Use melt (wide → long) and pivot (long → wide) to restructure it for your analysis.

# Wide → Long (melt)
# Before: columns = ['name', 'jan_sales', 'feb_sales', 'mar_sales']
df_long = df.melt(
    id_vars=['name'],
    value_vars=['jan_sales', 'feb_sales', 'mar_sales'],
    var_name='month',
    value_name='sales'
)

# Long → Wide (pivot)
df_wide = df_long.pivot_table(
    index='name', columns='month', values='sales', aggfunc='sum'
).reset_index()

# GroupBy aggregation
summary = df.groupby('category').agg(
    total_sales=('sales', 'sum'),
    avg_price=('price', 'mean'),
    num_orders=('order_id', 'count')
).reset_index()

Interview Questions