My Skill Lab
Data ScienceBeginner

Data Cleaning and Data Quality

Fix missing values, duplicates and dirty text — and keep data trustworthy.

Taught by Bibek Sharma · Data Scientist

What you'll learn

  • Profile any CSV or Excel export in pandas and produce an issue log ranked by impact
  • Diagnose why values are missing and choose between dropping, imputing and flagging
  • Remove exact and fuzzy duplicates and standardise categories with a versioned mapping
  • Convert messy text, dates and currency strings into correct types without silent data loss
  • Detect outliers with IQR and business rules, then encode those rules as a pandera schema
  • Reproduce the same cleaning in Excel and Power Query and keep data quality high over time

About this course

Dirty data quietly ruins dashboards, models and decisions. This beginner-friendly data cleaning course teaches you to find and fix missing values, duplicates, inconsistent categories, broken dates and messy text using pandas in Python, with the same techniques shown in Excel and Power Query so you can work wherever your data lives. You will profile a raw dataset, decide how to handle every gap, standardise text and dates, detect outliers, and write validation rules with pandera that catch bad records before they reach a report. Along the way you will learn the six dimensions of data quality that teams use to measure trust in their data. It is built for analysts, junior data scientists, operations staff and anyone who inherits spreadsheets and CSV exports. You need only basic Python or spreadsheet experience; every step is shown from scratch. You finish with a reusable, documented cleaning script and a checklist for keeping data quality high over time.

Course content

7 lessons
  1. 1Why Dirty Data Costs YouThe six dimensions of data quality, where dirty data comes from, and the cleaning workflow you will use throughout the course.Preview
  2. 2Loading and Profiling Data in pandasRead CSV and Excel files without losing information, then profile every column in a few lines of pandas.
  3. 3Missing Values: Find, Understand, DecideWhy data goes missing, how to see the pattern, and when to drop, fill or flag.
  4. 4Duplicates and Inconsistent CategoriesExact and fuzzy duplicates, choosing which record to keep, and collapsing 'UK', 'U.K.' and 'United Kingdom' into one.
  5. 5Cleaning Text, Dates and NumbersVectorised string methods, regex extraction, parsing dates safely and turning '£1,250.00' into a number.
  6. 6Outliers and Validation RulesSpot suspicious values with IQR and domain limits, then encode the rules as a pandera schema that fails loudly.
  7. 7Cleaning in Excel and Keeping Data TrustworthyThe same fixes in Excel and Power Query, plus the habits that stop the same mess coming back next month.