SlideShare a Scribd company logo

Getting started with Scrapy in Python

•
•
Viren Rajput
Viren Rajput

This document summarizes web scraping and introduces the Scrapy framework. It defines web scraping as extracting information from websites when APIs are not available or data needs periodic extraction. The speaker then discusses experiments with scraping in Python using libraries like BeautifulSoup and lxml. Scrapy is introduced as a fast, high-level scraping framework that allows defining spiders to extract needed data from websites and run scraping jobs. Key benefits of Scrapy like simplicity, speed, extensibility and documentation are highlighted.

Related slideshows

Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without ProgrammingAlmost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
 
Web Scraping and Data Extraction Service
Web Scraping and Data Extraction ServiceWeb Scraping and Data Extraction Service
Web Scraping and Data Extraction Service
 
Web Scraping Technologies
Web Scraping TechnologiesWeb Scraping Technologies
Web Scraping Technologies
 
1 of 11
Download to read offline
Web Scraping with Scrapy Virendra Rajput Hacker @Markitty
Agenda ● What is web scraping and why it's fun ● My experiments with web scraping ● Getting started with Scrapy ● How Scrapy works and a quick Demo ● Why Scrapy ● Questions
What is Web Scraping? ● Extracting information from websites ● Problem: ○ Static websites ○ No access to APIs to extract the data you need ○ Need to extract data periodically ● Manual solution - go to the website and copy the required data ● Smarter solution: Web Scraping
My Experiments with Scraping
Web Scraping in Python ● Download webpage with urllib2, requests ● Parse the page with BeautifulSoup/lxml ● Select with XPath or css selectors
Scrapy - fast high Level Screen Scraping and web crawling Framework ● Pick a website ● Define the data you want to scrape ● Write the spider to extract the data ● Run the spider ● Store the Data
Demo
Getting started with Scrapy in Python
Why Scrapy ● Simplicity ● Fast ● Productive/ Extensible ● Portable ● Well docs & Healthy community ● Commercial Support
Advanced Features (built in) ● Interactive shell for trying XPaths (useful for debugging) ● selecting and extracting data from html sources ● cleaning and sanitizing the scraped data ● generating feed exports (JSON, CSV) ● media pipeline for downloading stuff ● Middlewares for (cookies, HTTP compression, cache, user-agent spoofing, etc)
questions ?

More Related Content

Getting started with Scrapy in Python

  • 1. Web Scraping with Scrapy Virendra Rajput Hacker @Markitty
  • 2. Agenda ● What is web scraping and why it's fun ● My experiments with web scraping ● Getting started with Scrapy ● How Scrapy works and a quick Demo ● Why Scrapy ● Questions
  • 3. What is Web Scraping? ● Extracting information from websites ● Problem: ○ Static websites ○ No access to APIs to extract the data you need ○ Need to extract data periodically ● Manual solution - go to the website and copy the required data ● Smarter solution: Web Scraping
  • 5. Web Scraping in Python ● Download webpage with urllib2, requests ● Parse the page with BeautifulSoup/lxml ● Select with XPath or css selectors
  • 6. Scrapy - fast high Level Screen Scraping and web crawling Framework ● Pick a website ● Define the data you want to scrape ● Write the spider to extract the data ● Run the spider ● Store the Data
  • 9. Why Scrapy ● Simplicity ● Fast ● Productive/ Extensible ● Portable ● Well docs & Healthy community ● Commercial Support
  • 10. Advanced Features (built in) ● Interactive shell for trying XPaths (useful for debugging) ● selecting and extracting data from html sources ● cleaning and sanitizing the scraped data ● generating feed exports (JSON, CSV) ● media pipeline for downloading stuff ● Middlewares for (cookies, HTTP compression, cache, user-agent spoofing, etc)