Lecture 10 Notes — Social Network Analytics
Learning outcomes: LO3, LO4
Companion files: slides L10, notebook Week10_Social_Network_Analytics.ipynb, data follows.csv, traveller_profiles.csv
The follower network is synthetic: four interest communities of 150 travellers plus a few popular hubs, so the right answer for community detection is known.
1. Graphs
Nodes are travellers (600); directed edges are "follows" links (3,167); density 0.0088.
2. Degree and the long tail
Mean followers 5.3, median 4, maximum 88. Most people have few followers and a few have many, so the mean misleads. Hubs dominate how information moves.
3. Three centralities
| Measure | Question |
|---|---|
| Degree | Who has the most direct connections? |
| Betweenness | Who bridges separate groups? (expensive: sample it) |
| PageRank | Who is followed by important people? |
Spearman correlation between followers and PageRank is 0.76; 14 of the top 20 by followers are also in the top 20 by PageRank. Choose by the business question: reach, brokerage or endorsement.
4. Communities with Louvain
Louvain groups nodes to maximise modularity. It found 4 communities of 150 with modularity 0.62 and purity 1.00 against the true interests, without ever seeing the interest labels. Real networks are fuzzier: expect partial agreement.
5. PageRank
PR(v) = (1 − d)/N + d · Σ PR(u)/outdeg(u) over followers u of v, with damping d = 0.85 and dangling nodes sharing their rank. A manual NumPy version agrees with networkx to about 4 × 10⁻⁵.
On Spark RDDs each iteration is a join → flatMap → reduceByKey. Cache the link table and materialise the ranks every iteration, otherwise lineage grows and the job slows dramatically. Spark and NumPy results agree to under 10⁻⁴. Each iteration shuffles, which is why dedicated engines (GraphX, GraphFrames, Pregel) exist.
6. Information diffusion
An independent-cascade simulation (adoption probability 0.12, 300 runs per seed): a random traveller reaches about 3 people, the most-followed about 23, the top-PageRank traveller about 21. Reach depends strongly on the seed and on the adoption probability.
7. Limits
Sampling bias (you rarely see the full network), homophily versus influence, privacy (structure can re-identify users), and cost (all-pairs measures need sampling).
8. Discussion questions
- A hotel can seed a discount with one traveller. Which centrality would you use and why?
- How could you test whether communities reflect influence or similar tastes?
- Why does an iterative algorithm like PageRank suit Spark less well than a single
groupBy?
Next lecture: web analytics.