• Free

Web Scraping 101

  • Course
  • 24 Lessons

Learn to scrape all the things by building a Shopify App Store scraper.

Why should I learn to scrape?

When you need information for a work project or even a research task, you can't always find it "officially" with APIs or as downloadable content. 

Scraping, or "extracting data programmatically," can bridge this gap by automating up to millions of simple tasks on our behalf. Doing this not only provides a competitive edge through better informed decision-making, but it also can help us create magical user experiences.

In this course I'll demonstrate for you first-hand how to make scraping one of your most powerful weapons.

Curriculum

Welcome

Welcome to Web Scraping
What is scraping?
Scraping is a competitive advantage
Non-marketing use cases for scraping
[BONUS] Real world scraping case studies
Preview

Deciding what to scrape

Using the browser to find datapoints
Preview
Finding more specific elements
Summarizing data extraction challenges

Setting up your environment

Installing Ruby, Watir, and Webdrivers
Preview

Building our web scraper

Scraping and web development terminology
Scraping an app listing (Part 1)
Scraping an app listing (Part 2)
Fixing our Rating lookup
Using regex to sanitize results
Finding all app store listings
Scraping a sitemap with Nokogiri

Scraper upgrades

Upgrading to OOP (objected oriented programming)
Handling errors and anti-scraper workarounds

Using our scraper

Saving data to a spreadsheet
Scraping in a headless browser
Scraping effortlessly from the command line
[BONUS] Autosaving our results

Next steps

Three more performance improvement ideas
Request and thank you