Apache Spark and Big Data Processing
PySpark DataFrames, Spark SQL and distributed processing.
Taught by Sabina Tamang · Data Engineer
What you'll learn
- Explain how the driver, executors, lazy evaluation and Catalyst turn PySpark code into jobs, stages and tasks
- Read and write Parquet, CSV and Delta with explicit schemas and transform data with DataFrame and Spark SQL operations
- Count and reduce shuffles by reading physical plans and using adaptive query execution
- Choose and force the right join strategy and fix skewed joins with AQE and salting
- Tune partition counts, caching and executor memory, and diagnose slow stages from the Spark UI
- Build Structured Streaming pipelines with watermarks and checkpoints, use Delta Lake MERGE and time travel, and submit jobs to a cluster
About this course
When data outgrows one machine, Apache Spark is the engine most teams reach for, on Databricks, in the cloud or on a cluster of their own. This advanced course teaches PySpark the way it is used in production: DataFrames and Spark SQL rather than raw RDDs, an understanding of how the driver, executors and the Catalyst optimiser turn your code into stages and tasks, and the performance work that separates a job that finishes in minutes from one that fails at three in the morning. You will read and write Parquet at scale, execute joins that survive skewed keys, control partitions and caching, read the Spark UI to find spills and shuffles, and process streams with Structured Streaming and Delta Lake. It suits data engineers, machine learning engineers and analysts who already write confident Python and SQL and have worked with pandas or a warehouse. Everything runs on a laptop with pip install pyspark before it runs on a cluster.
Course content
7 lessons- 1How Spark Runs Your CodeDriver, executors and the cluster manager; lazy evaluation; jobs, stages and tasks; RDDs versus DataFrames.Preview
- 2DataFrames and Spark SQLRead and write Parquet and CSV, transform with column expressions, aggregate and window, and mix SQL with DataFrame code.
- 3Transformations, Shuffles and the OptimiserNarrow versus wide transformations, what a shuffle costs, reading explain() output, and adaptive query execution.
- 4Joins at ScaleSort-merge, shuffle-hash and broadcast joins, when Spark picks each, and salting to survive skewed keys.
- 5Partitions, Caching and MemoryInput, shuffle and output partitions; repartition versus coalesce; when caching helps and when it hurts; executor memory.
- 6Performance Tuning with the Spark UIA repeatable method for finding the slow stage, diagnosing spill, skew, small files and UDF cost, and fixing each.
- 7Streaming, Delta Lake and Production SparkStructured Streaming with watermarks and checkpoints, Delta Lake tables with ACID and time travel, and shipping jobs to a cluster.