Web Scraping with Python
requests, BeautifulSoup, Playwright and scraping ethically.
Taught by Anisha Rai · Senior Software Engineer
What you'll learn
- Judge whether a scraping task is permitted by reading robots.txt, terms of service and data-protection rules
- Fetch pages with requests sessions and parse HTML with BeautifulSoup CSS selectors
- Crawl paginated listings with stop conditions, deduplication and polite delays
- Scrape JavaScript-rendered pages with Playwright and locate hidden JSON APIs in DevTools
- Build resilient fetch functions with retries, backoff and 429 handling
- Store scraped records in SQLite with upserts and schedule the scraper to run unattended
About this course
Web scraping with Python is one of the most practical skills a data professional can have: prices, listings, reviews and public records all live on web pages, and this course teaches you to collect them reliably and responsibly. It is for analysts, developers and researchers who know basic Python and want to fetch pages with requests, parse HTML with BeautifulSoup, handle pagination, scrape JavaScript-rendered pages with Playwright, find hidden JSON APIs, and respect rate limits, robots.txt and the law. You will finish with a scheduled scraper that stores clean data in SQLite and copes with rate limits and transient errors. Along the way you learn how to read a site's terms, avoid personal data you should not collect, and keep your traffic polite. Prerequisites are Python basics, a terminal, and curiosity about how web pages are built.
Course content
6 lessons- 1How Scraping Works and When It Is AllowedWhat a scraper actually does, how to read robots.txt and terms, and a politeness checklist.Preview
- 2Fetching and Parsing With requests and BeautifulSoupMaking HTTP requests, inspecting HTML in DevTools and extracting fields with CSS selectors.
- 3Pagination and Crawling Many PagesFollowing next links, handling numbered pages and writing crawl loops that stop cleanly.
- 4JavaScript-Rendered Pages With PlaywrightWhy requests sees empty pages, and how to drive a real browser to get the rendered HTML.
- 5Hidden APIs, Rate Limits and RetriesCalling the JSON endpoints behind a page, and building fetch code that handles rate limits and errors.
- 6Storing, Cleaning and SchedulingChoosing a storage format, cleaning scraped fields and running the scraper unattended.