Scrapy workshop
Description If you want to get data from the web, and there are no APIs available, then you need to use web scraping! Scrapy is the most effective and popular choice for web scraping and is used in many areas such as data science, journalism, business intelligence, web development, etc. Abstract If you want to get data from the web, and there are no APIs available, then you need to use web scraping! Scrapy is the most effective and popular choice for web scraping and is used in many areas such as data science, journalism, business intelligence, web development, etc. This workshop will provide an overview of Scrapy, starting from the fundamentals and working through each new topic with hands-on examples. Participants will come away with a good understanding of Scrapy, the principles behind its design, and how to apply the best practices encouraged by Scrapy to any scraping task. Goals: Set up a python environment. Learn basic concepts of the Scrapy framework.
Related slideshows
Scrapy workshop
- 2. Karthik Ananth Who am I? ! Leading professional services @ Scrapinghub ! I have vision to synergise data generation and analytics ! Open source promoter
- 4. What is Web Scraping The main goal in scraping is to extract structured data from unstructured sources, typically, web pages.
- 5. What for ! Monitor prices ! Leads generation ! Aggregate information ! Your imagination is the limit
- 6. Do you speak HTTP? Headers, Query String Status Codes Methods Persistence GET, POST, PUT, HEAD… 2XX, 3XX, 4XX, 418 , 5XX, 999 Accept-language, UA*… Cookies
- 7. Standard Library HTTP for humans Let’s perform a request urllib2 python-requests
- 8. import requests req = requests.get('http://scrapinghub.com/about/') Show me the code! What now?
- 9. HTMLis not a regular language
- 10. lxml pythonic binding for the C libraries libxml2 and libxslt beautifulsoup html.parser, lxml, html5lib HTMLParsers
- 11. import requests import lxml.html req = requests.get(‘http://nyc2015.pydata.org/schedule/') tree = lxml.html.fromstring(req.text) for tr in tree.xpath('//span[@class="speaker"]'): name = tr.xpath('text()') url = tr.xpath('@href') print name print url Show me the code!
- 12. “Those who don't understand xpath are cursed to reinvent it, poorly.”
- 14. “An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.”
- 15. $ conda install -c scrapinghub scrapy
- 16. $ scrapy shell <url> An interactive shell console Invaluable tool for developing and debugging your spiders
- 17. An interactive shell console >>> response.url 'http://example.com' >>> response.xpath('//h1/text()') [<Selector xpath='//h1/text()' data=u'Example Domain'>] >>> view(response) # open in browser >>> fetch('http://www.google.com') # fetch other URL
- 18. $ scrapy startproject <name> pydata ├── pydata │ ├── __init__.py │ ├── items.py │ ├── pipelines.py │ ├── settings.py │ └── spiders │ └── __init__.py └── scrapy.cfg Starting a project
- 19. What is a spider
- 20. import scrapy class MySpider(scrapy.Spider): name = 'example.com' allowed_domains = ['example.com'] start_urls = [ 'http://www.example.com/', ] def parse(self, response): msg = 'A response from %s just arrived!' % response.url self.logger.info(msg) What is a Spider?
- 21. import scrapy class MySpider(scrapy.Spider): name = 'example.com' allowed_domains = ['example.com'] start_urls = [ ‘http://www.example.com/' ] def parse(self, response): for h3 in response.xpath(‘//h3/text()’).extract(): yield {‘title’: h3} for url in response.xpath('//a/@href').extract(): yield scrapy.Request(url, callback=self.parse) What is a Spider? 1.0
- 22. Batteries included ! Logging ! Stats collection ! Testing: contracts ! Telnet console: inspect a Scrapy process
- 23. Avoid getting banned ! Rotate your User Agent ! Disable cookies ! Randomized download delays ! Use a pool of rotating IPs ! Crawlera
- 24. A service daemon to run Scrapy spiders $ scrapyd-deploy Deployment 1.0 scrapyd
- 25. Scrapy Cloud $ shub deploy
- 27. Mandatory Sales Slide try.scrapinghub.com/pydatanyc Crawl the web, at scale • cloud-based platform • smart proxy rotator Get data, hassle-free • off-the-shelf datasets • turn-key web scraping
- 28. We’re hiring!
- 29. Thanks