Lecture 6 Notes — Predictive Analytics
Learning outcomes: LO3, LO4
Companion files: slides L06, notebook Week6_Predictive_Analytics.ipynb
1. Framing the problem
Task: predict whether a review gives 4 stars or more (high). Features must be known at prediction time: hotel stars, city, review length, helpful votes, number of tags, month. Anything computed from the rating is forbidden (leakage).
2. Baselines
Always predict the majority class first: accuracy 54.1%. For regression, predicting the mean gives RMSE 0.986. A model that cannot beat the baseline has learned nothing useful.
3. Comparing models with cross-validation
| Model | 5-fold CV AUC |
|---|---|
| Logistic regression | 0.592 |
| Random forest | 0.686 |
| Gradient boosting | 0.698 |
Cross-validation averages over several splits so one lucky split cannot mislead. The chosen model is then evaluated once on a held-out test set: accuracy 0.64, precision 0.58, recall 0.83, F1 0.68, AUC 0.69. The improvement over the baseline is genuine but modest: metadata alone cannot predict happiness precisely.
4. When a result is too good to be true
Adding the lexicon sentiment of the review text lifts AUC from 0.59 to 0.999. In our data this happens because the synthetic text was generated from the rating, so it leaks the answer by construction. Real text gives strong but never perfect signal. The habit to learn: when a result looks magical, assume leakage or a bug and investigate before celebrating.
5. The threshold is a business decision
A classifier outputs a probability; you choose the cut-off. At 0.3 recall is 99.9% but 80% of reviews are flagged; at 0.7 recall is 2.7%. Choose by the relative cost of a missed case and a false alarm, not by the default 0.5.
6. Overfitting
A deep decision tree reaches train AUC 0.985 but validation AUC 0.625. The gap between training and validation scores signals memorisation. Remedies: simpler models, more data, regularisation, early stopping.
7. Data leakage
- Target-derived feature: the hotel's mean rating including the review being predicted gives AUC 0.68, versus 0.66 for the honest leave-one-out version.
- Group leakage: when the same hotel appears in train and test the model can recognise the hotel (random split AUC 0.662 versus grouped split 0.646). Split the way the model will be used.
8. Regression and explanation
Gradient boosting reduces RMSE from 0.986 to 0.875. Permutation importance (shuffle a feature and measure the loss) ranks review length (n_words) first. Importance describes the model, not causes in the world.
9. Spark MLlib
A Pipeline of VectorAssembler and GBTClassifier trains on a cluster; on metadata only it scored AUC 0.585, consistent with the single-machine model. The discipline (baseline, validation, leakage checks) does not change with scale.
10. Discussion questions
- Which costs more for a hotel: missing an unhappy guest or flagging a happy one? What threshold follows?
- Why is a grouped split the honest choice for predicting reviews of new hotels?
- A teammate reports AUC 0.99. List three things you would check.
Next lecture: prescriptive analytics.