Lecture 5 Notes — Descriptive Analytics
Learning outcome: LO3 (analytics on structured data)
Companion files: slides L05, notebook Week5_Descriptive_Analytics.ipynb, data reviews_clean.csv
1. Four kinds of analytics
| Type | Question | Weeks |
|---|---|---|
| Descriptive | What happened? | 5 |
| Predictive | What will happen? | 6 |
| Prescriptive | What should we do? | 7 |
| Specialised | Survival, text, network, web | 8–11 |
Descriptive analytics is the foundation: every model rests on understanding what the data looks like.
2. Centre, spread and shape
On the 15,183 rated reviews: mean 3.60, median 3.5, standard deviation 0.99, quartiles 3.0 / 3.5 / 4.5, skewness −0.35 (a longer tail of low ratings). The mean and median are close, so the distribution is only mildly skewed. Always report centre and spread together, and plot before summarising: ratings are bounded (1–5) and come in half-stars, which a histogram shows and a mean hides.
3. Aggregation and pivots
groupby is the workhorse. Mean rating by city runs from Colombo 3.49 to Ella 3.70: a gap of only 0.21 stars against a standard deviation of 0.99. The difference may be real, but ask whether it is large enough to change a decision. A pivot table (city × star class) with a heatmap lets the eye find patterns in a matrix.
4. Time trends
Monthly mean rating stays between 3.5 and 3.7. Movement of that size is mostly noise: check the number of reviews per month before telling a story, and use a rolling mean to see the trend rather than the jitter.
5. Correlation is not causation
Rating and text sentiment correlate strongly (r = 0.86); rating and hotel star class barely correlate (r = −0.05). Correlation measures linear co-movement only. It cannot show cause, and it is blind to non-linear patterns.
6. The small-sample trap in rankings
Ranking hotels by raw mean rewards hotels with few reviews, because small samples are noisy. A shrinkage (Bayesian) average pulls small samples toward the global mean C:
score = (n · mean + m · C) / (n + m), here with m = 20.
Our hotels all have at least about 30 reviews, so the effect is mild (median reviews in the top 10: 49.5 naive, 58 shrunk). In a simulation where each hotel has only 3–60 reviews, the shrinkage ranking recovered 5.9 of the true top 10 versus 5.3 for the naive ranking. When every hotel has the same n, shrinkage cannot change the order. Rule: show n beside every score.
7. Descriptive analytics at scale
groupBy().agg() is a wide transformation (shuffle). Exact quantiles are expensive in a distributed system, so Spark offers approxQuantile with a relative-error parameter; with error 0.01 it returned the same quartiles (3.0, 3.5, 4.5) as pandas.
8. Common mistakes
- Averaging without looking at the distribution or the group sizes.
- Reading small differences between groups as meaningful.
- Reporting a trend from months with very different review counts.
- Treating correlation as cause.
9. Discussion questions
- When would you report the median instead of the mean for review ratings?
- A manager sees Ella (3.70) beating Colombo (3.49) and wants to move budget. What would you check first?
- Why can a ranking be "correct" and still unfair to a new hotel?
Next lecture: predictive analytics.