Course Overview

What you will learn

SENG 42293 Big Data Analytics builds analytics on top of the Hadoop/Spark infrastructure from SENG 41303 Big Data Infrastructure. You will learn to reason about the characteristics of big data, collect and sample it responsibly, make it trustworthy, and then analyse it with descriptive, predictive and prescriptive methods, survival analysis, text and sentiment analytics, social-network analytics, web analytics and visualisation.

Four capabilities run through the course: understanding big data and its opportunities and challenges; engineering data quality for structured and semi-structured sources; choosing and applying the right analytical method; and operating on multi-node clusters with Spark.

3 Parts · 12 Lectures · Weekly Labs · 1 Semester Project

Assessment: end-of-semester examination (70%), travel-review analytics pipeline project (20%), concepts quiz (10%). Weekly labs are practice for the exam and project and carry no marks.

LO1

Describe the characteristics of big data.

Lectures 01 · 02
LO2

Explain the opportunities and challenges of big data.

Lecture 01
LO3

Apply analytics techniques to structured and semi-structured data.

Lectures 02 – 12
LO4

Work with multi-node clusters for large-scale analytics.

Lectures 04 · 12 · Pipeline Guide

Build a complete analytics pipeline

Raw nested JSON → quarantine → flatten & clean → bronze/silver/gold Parquet → features → analytics → charts, then scale to a multi-node Spark cluster. Tested end to end, with diagrams, expected outputs and troubleshooting.

I

Foundations

Lectures 01 – 04 · What big data is, how it is collected, and how to make it trustworthy

Course Introduction & Big Data Characteristics
The five V's, opportunities and challenges
L01 · Part I · LO1, LO2
01
  • Traditional data vs big data: why single-machine tools break down
  • Volume, Velocity, Variety, Veracity, Value (and Variability)
  • Business and technical opportunities; technical, business and ethical challenges
  • Case discussion: Netflix recommendations
  • Course roadmap and the semester project
Data Collection & Sampling
Batch vs streaming, sampling designs, bias, and stream algorithms
L02 · Part I · LO1, LO3
02
  • Batch vs streaming collection; choosing a technique
  • Random, stratified, systematic and cluster sampling with worked examples
  • Selection, survivorship, non-response and undercoverage bias; Google Flu Trends
  • Sample size n = Z²p(1−p)/e² (n ≈ 385) and where 1.96 comes from
  • Reservoir sampling, Count-Min Sketch, HyperLogLog
Data Quality & Pre-processing I
Quality dimensions, missing data, duplicates, outliers, encoding, scaling
L03 · Part I · LO3
03
  • Six data quality dimensions with concrete checks
  • MCAR / MAR / MNAR and imputation strategies
  • Exact vs near-duplicate detection; IQR and z-score outliers
  • One-hot vs ordinal encoding; min-max vs z-score scaling
  • A reusable cleaning pipeline (311 → 288 rows in the notebook)
Data Quality & Pre-processing II
Semi-structured data, schema-on-read, and PySpark on a cluster
L04 · Part I · LO3, LO4
04
  • JSON and XML: anatomy, nesting, nulls; schema-on-write vs schema-on-read
  • Flattening nested data with explode; type drift and corrupt records
  • Spark architecture: driver, cluster manager, executors; lazy evaluation; shuffles
  • Week 3 pipeline translated to PySpark; Parquet; feature engineering at scale
  • In-class word-count activity (Map → Shuffle → Reduce)
II

Analytics Core

Lectures 05 – 08 · Describing, predicting, prescribing, and modelling time-to-event

Descriptive Analytics
Summarising and exploring data at scale
L05 · Part II · LO3
05
  • Centre, spread and shape; group-by, pivots and heatmaps
  • Trends, and correlation is not causation
  • The small-sample trap: shrinkage rankings
  • KPI dashboard and Spark approxQuantile
Predictive Analytics
Regression, classification, and honest evaluation
L06 · Part II · LO3
06
  • Baselines, cross-validation and held-out testing
  • Thresholds, precision, recall and AUC
  • Overfitting and data leakage (random vs grouped splits)
  • Permutation importance and Spark MLlib pipelines
Prescriptive Analytics
From predictions to decisions
L07 · Part II · LO3
07
  • Decision variables, objective, constraints (MILP)
  • Naive vs greedy vs optimal plans
  • What-if analysis, frontier and shadow prices
  • Monte Carlo robustness and overbooking
Survival Analysis
Modelling time until an event
L08 · Part II · LO3
08
  • Censoring and why simple averages mislead
  • Kaplan–Meier built by hand; log-rank test
  • Hazard and the Cox model; checking assumptions
  • Retention what-if and a Spark life table
III

Specialised Analytics & Visualisation

Lectures 09 – 12 · Text, networks, the web, and communicating results

Sentiment & Text Analytics
From raw text to features and sentiment
L09 · Part III · LO3
09
  • Pre-processing, bag of words, TF-IDF, n-grams
  • Lexicon vs learned sentiment; negation failures
  • NMF topics and aspect scorecards
  • Spark ML text pipeline
Social Network Analytics
Graphs, centrality and communities
L10 · Part III · LO3
10
  • Degree, betweenness and PageRank
  • Louvain communities checked against ground truth
  • PageRank by hand, then on Spark RDDs
  • Information diffusion simulation
Web Analytics
Clickstreams, sessions, funnels and experiments
L11 · Part III · LO3
11
  • Sessionisation (30-minute rule), metrics, funnel
  • A/B z-test, confidence interval, power
  • The peeking problem (A/A simulation)
  • Sessionisation with Spark window functions
Visualisation & the End-to-End Pipeline
Communicating results; running the pipeline on a multi-node cluster
L12 · Part III · LO3, LO4
12
  • Misleading vs honest charts; choosing a chart
  • Small multiples and a one-screen dashboard
  • Pipeline quality gates (7 checks)
  • Capstone report structure and assessment

Assessments

Labs, project & examinations

Weekly Labs · Individual

Hands-on Notebook Labs

Weekly · Jupyter + PySpark (or Databricks Community Edition)

Each lecture has a practical: the sampling notebook, the data-quality pipeline, the word-count activity, and the pipeline lab. Not graded: they are the practice that prepares you for the project and the examination, so attempt every one and compare your results with the solutions discussed in class.

  • Run every cell; explain what each result means
  • Answer the discussion questions in your own words
  • Ask for feedback in class or office hours
Formative · not graded

Semester Project

Travel-Review Analytics Pipeline

Continuous · built up week by week · demonstrated at the end

Teams build an end-to-end analytics pipeline on a travel-review dataset, following the Pipeline Guide and adding a new analytics layer each stage. Use a licensed public dataset or an API whose terms allow reuse.

  • M1 (after L02) — data source and sampling plan
  • M2 (after L04) — ingest, flatten, clean, store as Parquet
  • M3 (after L06) — descriptive dashboard and a first predictive model
  • M4 (after L09) — text and sentiment features
  • M5 (after L12) — full pipeline on a multi-node cluster + demo and report
20% of Final Grade

Mid-Semester Quiz

Concepts Quiz

Mid-semester · covers Part I (Lectures 01 – 04)

Short-answer and multiple-choice questions on big-data characteristics, sampling, data quality and Spark's execution model.

  • Classify examples by the V's; spot sampling bias
  • Choose imputation and outlier strategies for a scenario
  • Explain narrow vs wide transformations
10% of Final Grade

Final Examination

End-of-Semester Examination

Examination Period · Covers all 12 lectures

Tests application and judgement: choosing methods, interpreting outputs, and designing a pipeline for a stated problem, rather than recalling definitions.

  • Part I — collection, sampling, quality, Spark
  • Part II — descriptive, predictive, prescriptive, survival
  • Part III — text, network, web, visualisation
70% of Final Grade

Your Journey

Learning path through this course

Weeks 1–2
Establish foundations. Learn the five V's, then how data is collected and sampled. Run the sampling notebook and check the bias examples against your own project data source.
Weeks 3–4
Make data trustworthy. Work through the cleaning pipeline and the semi-structured data lecture. Do the word-count activity, then run the Pipeline Guide through step 6.
Weeks 5–8
Analyse. Descriptive, predictive and prescriptive analytics, then survival analysis. Extend your project pipeline with a dashboard and a first model.
Weeks 9–11
Specialise. Text and sentiment, social networks, and web analytics. Replace the toy sentiment lexicon in the pipeline with real text features.
Week 12
Synthesise. Visualise and communicate. Run the whole pipeline on a multi-node cluster (Pipeline Guide step 7) and present your project.
Exam Period
Final examination. Practise explaining your pipeline decisions: why this sampling design, this imputation, this storage format.

Course Resources

Materials & references

🚀

Pipeline Guide

Step-by-step capstone with diagrams, code and checkpoints

📦

Pipeline Project Files

Runnable code, sample data, Docker Compose cluster

📓

Week 2 Notebook

Sampling, reservoir, Count-Min, HyperLogLog

📓

Week 3 Notebook

Data-quality pipeline on the messy-customers data

⚡

Spark Documentation

SQL, DataFrame and ML guides; cluster mode overview

🧱

Apache Parquet

Columnar storage format used throughout the pipeline

🐳

Docker Docs

Install Docker and use Compose for the cluster lab

📘

Suggested Reading

Damji et al., Learning Spark, 2nd ed. · Kleppmann, Designing Data-Intensive Applications · Leskovec et al., Mining of Massive Datasets

About This Module

Module Information

SENG 42293 — Big Data Analytics is a 3-credit Level 4 compulsory module delivered by Eng. Dr. Tiroshan Madushanka of the Software Engineering Teaching Unit, University of Kelaniya.

Pre-requisite: SENG 41303 Big Data Infrastructure. You should be comfortable with the Hadoop/Spark ecosystem, distributed storage concepts and basic cluster operations.

Academic integrity: labs, notebooks and project reports are checked for plagiarism. Write explanations in your own words.

Software to install: Python 3.10–3.13, Jupyter, Java 17, PySpark 3.5 (or Databricks Community Edition), and Docker Desktop for the cluster lab.

Contact: Office hours by appointment. Use the University LMS for submissions and announcements.