Lecture 3 Notes — Data Quality & Pre-processing I
Companion files: slides L03, notebook Week3_Data_Quality_Preprocessing.ipynb, dataset messy_customers.csv
1. Recap & Orientation (5–10 min)
- Quick recap of Week 2: data sources, collection methods, sampling theory (classical + big-data scale), sampling bias
- Bridge: "You now have a way to get a representative dataset. This week is about making sure that data is actually usable before any analysis touches it."
- Today's agenda: what data quality means, missing data, duplicates & noise, outliers, encoding, and scaling — then a hands-on cleaning lab
2. What Is Data Quality, and Why It Matters (15–20 min)
The cost of bad data - Open with a framing question: "If your dataset is wrong, does it matter how good your model is?" - Real-world cost examples worth citing: IBM's oft-quoted estimate that poor data quality costs the US economy over $3 trillion/year; a flipped sign or a duplicated record silently doubling a KPI; a machine learning model trained on biased or corrupted data making systematically wrong decisions - Tie back to Week 1's Veracity (the "V" of big data) — this week is where Veracity gets operationalized
The data quality dimensions — maps to LO2/LO3 - Completeness — is anything missing that should be there? - Accuracy — does the data correctly reflect reality? - Consistency — does the same fact agree across records/sources (e.g., "USA" vs "United States" vs "U.S.")? - Timeliness — is the data current enough to be useful? - (Worth mentioning as a bonus pair): Validity (does it conform to a defined format/range?) and Uniqueness (no unintended duplicates)
In-class activity (5 min): Show a small messy sample table (a few rows with a missing age, a duplicated customer, an "age = 250", inconsistent country names, and a stale timestamp). Ask students to call out which dimension each problem violates. This sets up everything that follows.
3. Missing Data (20–25 min)
Why data goes missing - Sensor failure, skipped survey questions, failed API calls, deliberate non-disclosure, merge/join mismatches
Types of missingness (important conceptually, even if briefly covered) - MCAR (Missing Completely At Random) — missingness unrelated to any value - MAR (Missing At Random) — missingness related to other observed variables (e.g., older respondents skip an income question) - MNAR (Missing Not At Random) — missingness related to the missing value itself (e.g., people with very high incomes decline to answer) - Why this matters: the type of missingness determines whether a simple fix is safe or whether it will bias your results — ties directly back to Week 2's non-response bias
Detecting missing data
- df.isnull().sum(), visualizing missingness patterns (e.g., a missingness heatmap)
Handling strategies
- Deletion: listwise (drop the row) vs pairwise — simple, but wastes data and can bias results if not MCAR
- Imputation:
- Simple: mean/median (numeric), mode (categorical)
- Time-aware: forward-fill / backward-fill for time series
- Model-based: regression or k-NN imputation (brief mention — "more accurate, more expensive")
- Flagging: adding an is_missing indicator column so the model can learn from the missingness itself
- Key teaching point: there is no universally correct method — the right choice depends on why the data is missing and how much of it there is
4. Duplicates & Noisy Data (15 min)
Duplicates
- Exact duplicates — easy: df.duplicated()
- Fuzzy/near duplicates — the harder, more realistic case: "John Smith" vs "Jon Smith", same transaction logged twice with slightly different timestamps
- Brief mention of fuzzy matching (e.g., Levenshtein distance, fuzzywuzzy/rapidfuzz) as a pointer for later, not a deep dive
Noisy data - Random errors, sensor jitter, typos, inconsistent units (kg vs lbs mixed in one column) - Basic smoothing ideas: binning, moving averages (light touch — full treatment is more of a signal-processing topic)
5. Outliers (15–20 min)
What counts as an outlier — and why it's not always "bad data" - Could be an error (typo: age = 250) or a genuinely rare but real event (a fraud transaction, a billionaire's income) — the handling strategy depends on which
Detection techniques - IQR method: flag anything outside Q1 − 1.5×IQR to Q3 + 1.5×IQR - Z-score method: flag |z| > 3 (ties back nicely to last week's z-score chart) - Visual: boxplots, scatter plots
Handling strategies - Remove (only if clearly an error), cap/winsorize, transform (log transform for skewed data), or leave as-is and flag for a robust model downstream
6. Encoding Categorical Variables (10–15 min)
- Nominal (no order: color, country) → one-hot encoding
- Ordinal (has order: low/medium/high) → ordinal/label encoding preserving rank
- Brief mention of target encoding for high-cardinality categories (e.g., hundreds of cities) — flagged as "useful, but easy to leak information if done carelessly, more on this in Week 6"
- Common pitfall to flag explicitly: one-hot encoding a column with 500 categories silently exploding your feature space
7. Normalization & Standardization (10–15 min)
- Min-max scaling: rescales to [0,1] — sensitive to outliers
- Z-score standardization: mean 0, std 1 — the more common default, especially after this week's outlier discussion
- When it matters: distance-based and gradient-based methods (k-NN, k-means, neural nets, regression with regularization) need it; tree-based methods (random forests, gradient boosting) generally don't
- Quick rule of thumb for students: "If your model computes a distance or a gradient, scale your features."
8. Hands-On Lab (45–60 min)
Build a cleaning pipeline in Python (pandas)
1. Load a deliberately messy sample dataset (missing values, duplicates, inconsistent categories, a few extreme outliers)
2. Profile it: .info(), .isnull().sum(), .describe(), value counts on categoricals
3. Handle missing values — justify the method chosen per column, not just applying one rule everywhere
4. Remove/resolve duplicates
5. Detect and handle outliers (IQR or z-score) on at least one numeric column
6. Encode categorical variables appropriately (one-hot vs ordinal, based on the variable)
7. Standardize the numeric features
8. Deliverable: a short notebook with each step justified in a markdown cell ("why did you choose mean over median here?") — the reasoning matters as much as the code
(If you'd like, this is also a natural point to plug in the continuous course project — the same cleaning pipeline applied to the travel-review dataset once it's collected, rather than a generic toy dataset. Happy to adjust the lab around that if you want to lock that in now.)
9. Wrap-Up & Next Week (5 min)
Today's key takeaways - Data quality has concrete, checkable dimensions — not just a vague "is it good?" - Missing data isn't one problem — MCAR/MAR/MNAR changes what's safe to do about it - Outliers require judgment, not a blanket rule - Scaling and encoding choices depend on what's downstream, not on habit
Next lecture (Lecture 4): Data Quality & Pre-processing II - Same problems, but at scale: semi-structured data (JSON/XML), schema-on-read vs schema-on-write, and doing all of this in PySpark on a multi-node cluster instead of a single pandas DataFrame