Lecture 11 Notes — Web Analytics
Learning outcomes: LO3, LO4
Companion files: slides L11, notebook Week11_Web_Analytics.ipynb, data clickstream.csv
The clickstream is synthetic, with a built-in lift for variant B so the experiment has something to find. Real effects are usually smaller and noisier.
1. Events and sessions
44,276 events from 6,000 visitors. A session starts when the gap since a visitor's previous event exceeds 30 minutes (a convention). Sort by user and time, compute the gap, flag new sessions, and take a running sum. Result: 10,084 sessions (1.68 per visitor). In this synthetic log the count is identical at 10, 30 and 60 minutes because gaps are either minutes or hours; real logs are messier. State the rule you used.
2. Core metrics
Bounce rate (single-event sessions) 14.7%; session conversion 12.5%; 4.39 pages per session. Desktop converts best (14.5%), mobile lowest (11.1%); channels differ little (12.2–12.6%).
3. The funnel
Distinct visitors per step: home 6,000 → search 5,508 → hotel page 5,508 → booking start 2,172 → payment 1,436 → confirmation 1,170. The largest relative drop is into booking start (39% of hotel-page visitors continue). Count distinct visitors, not events.
4. A/B testing
Visitors are randomised to A or B. Analyse by visitor, the randomisation unit. With pooled rate p̂:
z = (p̂_B − p̂_A) / √( p̂(1 − p̂)(1/n_A + 1/n_B) )
Result: A 18.2% versus B 20.8%, lift +2.5 points (+13.9% relative), z = 2.48, p = 0.013, 95% CI for the lift +0.5 to +4.5 points. The lift appears on every device (mobile +3.2 points). Significance is not business value: read the interval.
5. Power and sample size
n per group ≈ 2 p̄(1 − p̄)(z_{α/2} + z_β)² / δ². At a baseline near 18%, detecting a 3-point lift needs about 2,800 visitors per group; a 1-point lift about 23,900; a 0.5-point lift about 94,600. Halving the effect roughly quadruples the sample. Plan before you run.
6. The peeking problem
In an A/A simulation (identical designs) testing once at the end gives 4.7% false positives, but stopping at the first significant daily peek gives 28.2%. Fix the sample size in advance, or use sequential methods designed for monitoring.
7. Spark
Sessionisation with lag() over a window partitioned by user, plus a running sum(), reproduces the 10,084 sessions. Production systems run this incrementally over hourly Parquet partitions.
8. Pitfalls
Bots, wrong unit of analysis, early stopping, novelty effects, sample-ratio mismatch, multiple comparisons across segments, privacy of identifiers.
9. Discussion questions
- Design a test for a 0.5-point lift: how many visitors, and how many days at 6,000 visitors?
- Why might mobile show the biggest lift, and how would you check it is not chance?
- What would you do if the split came out 55/45?
Next lecture: visualisation and the capstone.