SENG-42293

Travel-Review Big Data Pipeline (SENG 42293)

Raw nested JSON -> quarantine bad lines -> flatten & clean -> Parquet (bronze/silver/gold) -> features -> analytics -> charts.

Quick start (local, no Docker)

python3 -m venv .venv && source .venv/bin/activate   # Python 3.10-3.13 work with Spark 3.5 (setuptools is in requirements.txt)
pip install -r requirements.txt                       # needs Java 17 (or 11/21) installed
./run_all.sh

Outputs: lake/ (Parquet layers), output/ (CSV summaries + PNG charts).

Multi-node run (Docker)

./run_cluster.sh        # master + 2 workers, then open http://localhost:8080

The compose file was not executed in the authoring environment - test it before class.

Layout

• data/generate_reviews.py - seeded generator of messy nested JSON (data/raw/ is already generated)
• pipeline/01_ingest.py ... 05_dashboard.py - one script per stage; common.py holds paths + SparkSession
• docker-compose.yml, run_all.sh, run_cluster.sh

The data is synthetic. Replace data/raw/ with a licensed real dataset (e.g. a public hotel-review dump) for the semester project; respect site terms before scraping anything.