Lecture 1 Notes — Course Introduction & Big Data Characteristics
Learning outcomes: LO1 (big-data characteristics), LO2 (opportunities and challenges)
Companion files: slides L01
1. Course Orientation (15 min)
- Welcome, course code/credits, position in the Data Science specialization
- Link to prerequisite: SENG 41303 (Big Data Infrastructure) — briefly recap what students should already know (Hadoop/Spark ecosystem, distributed storage, cluster basics) since this course builds analytics on top of that infrastructure
- Assessment overview: labs/assignments, group project (spanning the semester), quiz, final exam
- Tools students will need installed: Python 3.x, Jupyter, Pandas, PySpark (or Databricks Community Edition access)
2. What is Big Data? (20 min)
- Traditional data processing vs big data — why relational databases and single-machine tools break down
- The Characteristics of Big Data (the "V's") — maps to LO1
- Volume — scale of data generated (examples: social media, IoT sensors, transaction logs)
- Velocity — speed of generation/processing (batch vs real-time streaming)
- Variety — structured (tables), semi-structured (JSON, XML, logs), unstructured (text, images, video)
- Veracity — uncertainty/quality of data; noise, inconsistency, incompleteness
- Value — the actual business insight extracted; the point of everything else
- (Optional 6th V: Variability — inconsistency in data flow/meaning over time)
- Short in-class discussion: ask students to classify examples (bank transactions, tweets, sensor streams, hospital records) by which V's dominate
3. Business & Technical Opportunities and Challenges (30 min) — maps to LO2
Opportunities: - Personalization (recommendation engines), fraud detection, predictive maintenance, customer sentiment tracking, operational efficiency, new business models built entirely on data (e.g. ride-sharing, streaming platforms)
Challenges: - Technical: storage/compute scaling, data integration across formats, latency requirements, data quality, security/privacy, skills gap - Business: cost of infrastructure, data governance, regulatory compliance (GDPR-type concerns), organizational resistance to data-driven decision-making, ethical use of data (bias, surveillance)
Case discussion (10–15 min): Pick one real example (e.g., Netflix recommendations, a bank's fraud detection system) and walk through which V's apply and which opportunities/challenges show up.
4. Where This Course Fits (10 min)
- Preview of the semester: from raw data → cleaning → descriptive → predictive → prescriptive → specialized analytics (text, social network, web) → visualization
- Explain the semester spine: one continuous travel-review project grows each week, ending in an end-to-end pipeline on a multi-node cluster (see the Pipeline Guide)
Suggested close: short formative quiz (5 questions) on the V's and opportunity/challenge classification, or a 2-minute reflection write-up.