My Skill Lab
Data ScienceIntermediate

Data Engineering Fundamentals

Pipelines, warehouses, orchestration and the modern data stack.

Taught by Sabina Tamang · Data Engineer

What you'll learn

  • Map the data lifecycle and the modern data stack, and choose between batch, micro-batch and streaming
  • Extract data incrementally from databases and APIs, land it as Parquet, and explain change data capture
  • Choose between a data lake, a cloud warehouse and a lakehouse table format, and partition tables for cost
  • Build layered, tested and documented SQL transformations with dbt, including incremental models
  • Design star schemas with declared grain and track history with type 2 slowly changing dimensions
  • Orchestrate idempotent pipelines with Apache Airflow and operate them with quality checks, lineage and cost controls

About this course

Data engineering is the fastest-growing, best-paid corner of the data world, and it is where analysis becomes reliable. This intermediate course explains how modern data platforms are built and lets you build a small one yourself. You will learn the data lifecycle from ingestion to serving, the difference between batch and streaming, and why ELT has replaced ETL for most teams. You will land data in a lake or a cloud warehouse such as BigQuery or Snowflake, transform it with SQL and dbt, model it into star schemas with slowly changing dimensions, orchestrate it with Apache Airflow, and protect it with data quality tests and observability. Every lesson uses tools you can run for free on a laptop. It suits analysts moving into engineering, backend developers joining a data team and anyone who owns a pipeline and wants to own it well. You need comfortable SQL and basic Python. You finish with a working end-to-end pipeline and the vocabulary to design the next one.

Course content

7 lessons
  1. 1What Data Engineers Actually BuildThe data lifecycle, the modern data stack, and batch versus streaming, explained with one company's pipeline.Preview
  2. 2Ingestion: Batch Loads, APIs and Change Data CaptureExtract from databases, APIs and files reliably, land raw data as Parquet, and understand CDC and Kafka for near-real-time feeds.
  3. 3Storage: Lakes, Warehouses and LakehousesObject storage and Parquet, cloud warehouses such as BigQuery and Snowflake, open table formats, and partitioning for cost.
  4. 4Transformation with SQL and dbtELT in practice: layered SQL models, ref() dependencies, incremental models, tests and documentation with dbt.
  5. 5Data Modelling: Star Schemas and Changing DimensionsFacts, dimensions and grain, the star schema analysts love, and tracking history with type 2 slowly changing dimensions.
  6. 6Orchestration with Apache AirflowDAGs, tasks, schedules and sensors; idempotent, backfillable pipelines with the TaskFlow API.
  7. 7Data Quality, Observability and CostFreshness, volume and schema checks, lineage and alerting, plus the habits that keep warehouse bills predictable.