Crawl Budget Conqueror - Take Control of Your Crawl Budget
•
2 likes•265 views
Jori Ford presented on how to take control of a website's crawl budget. She explained that crawl budget is determined by crawl rate and demand. Crawl rate depends on factors like response time and server errors, while demand is based on popularity and freshness. She recommended quantifying a site's current crawl budget by analyzing log files to see what pages are crawled and how often. Then pages can be optimized by improving response time and removing errors. Non-essential pages should be pruned to conserve budget, while important pages can be better linked and updated to increase demand. Overall the key is understanding how Google allocates crawl resources and managing page quality and relevance accordingly.
Report
Share
Crawl Budget Conqueror - Take Control of Your Crawl Budget
- 1. Jori Ford | @chicagoseopro | #TechSEOBoost #TechSEOBoost | @CatalystSEM THANK YOU TO THIS YEAR’S SPONSORS Crawl Budget Conqueror - Take Control of Your Crawl Budget Jori Ford, G2
- 2. Jori Ford | @chicagoseopro | #TechSEOBoost Take Control of Your Crawl Budget! Crawl Budget Conqueror
- 3. Jori Ford | @chicagoseopro | #TechSEOBoost Agenda How Does Google See Crawl Budget? Key Factors to Managing Crawl Budget Ways to Quantify Crawl Budget on Your Site Fine-Tuning & Crawl Controls • Conservation - Tactics to help you maximize the crawl budget you have • Optimization - Shifting crawl to places you want • Expansion - Get More, Why Not!
- 4. Jori Ford | @chicagoseopro | #TechSEOBoost How Does Google See Crawl Budget?
- 5. Jori Ford | @chicagoseopro | #TechSEOBoost Deconstructing Crawl Budget Crawl Budget = Crawl Rate x Crawl Demand
- 6. Jori Ford | @chicagoseopro | #TechSEOBoost Crawl Rate is Variable “If the site responds really quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Googlebot crawls less.” -- Gary Illyes, Google
- 7. Jori Ford | @chicagoseopro | #TechSEOBoost Crawl Rate Breakdown = {Server Response Time x Time} Error Rate Result: = {ttfb x duration} %server error
- 8. Jori Ford | @chicagoseopro | #TechSEOBoost Crawl Demand = Popularity x Freshness
- 9. Jori Ford | @chicagoseopro | #TechSEOBoost Popular on the Internet =
- 10. Jori Ford | @chicagoseopro | #TechSEOBoost Staleness aka Freshness In general, we try to do our crawling based on what we think this page might be changing or how often it might be changing. So, if we think that something stays the same for a longer period of time, we might not crawl it for a couple of months. -- John Mueller, Google
- 11. Jori Ford | @chicagoseopro | #TechSEOBoost Crawl Budget = {Avg. ttfb x duration/%server error} x {(CTR) x (Avg Time b/t Updates)} Key Factors to Managing Crawl Budget: • Time to First Byte • Server Error Rate • Click-Thru Rate (CTR) • Freshness
- 12. Jori Ford | @chicagoseopro | #TechSEOBoost Ways to Quantify Crawl Budget on Your Site
- 13. Jori Ford | @chicagoseopro | #TechSEOBoost Quantifying Crawl Budget on Your Site Step 1: Determine how many pages you’re expecting to be crawled. Step 2: Figure out what pages Google is crawling? Step 3: How many are there? And with what frequency are they crawled? Step 4: Segment your data by page type.
- 14. Jori Ford | @chicagoseopro | #TechSEOBoost Step 1: Determine # of Pages Expected 1. What are your money pages? 2. Group by Page Types or Templates 3. Match to XML Sitemaps (ONLY if optimized)
- 15. Jori Ford | @chicagoseopro | #TechSEOBoost Step 2: What Pages are Being Crawled? • Raw Server Crawl Logs • Log File Analyzer Tools (Botify, OnCrawl, Screaming Frog, Deep Crawl) • Custom Tracking (GTM, Cloud Service Workers)
- 16. Jori Ford | @chicagoseopro | #TechSEOBoost Step 3: # of Crawled Pages & Frequency • Excel Using Pivot Tables • SQL Database Query • Log File Analyzer Tools (Botify, OnCrawl, Screaming Frog, Deep Crawl)
- 17. Jori Ford | @chicagoseopro | #TechSEOBoost Step 4: Segment by Type % Page Type % Crawl Allocation % Active vs. Inactive % Not Crawled
- 18. Jori Ford | @chicagoseopro | #TechSEOBoost Calculate Your Crawl Budget = (Avg # of Crawled URLs x Freq) Time
- 19. Jori Ford | @chicagoseopro | #TechSEOBoost Fine-Tuning & Crawl Controls
- 20. Jori Ford | @chicagoseopro | #TechSEOBoost Conserve Your Crawl Budget • Prune Non-Money Pages • De-Index Dupes • Remove pages not in XML Sitemap(s) • Removal or fixing of Erroring Pages
- 21. Jori Ford | @chicagoseopro | #TechSEOBoost Optimize Crawling • Improve TTFB (time to first byte) • Optimize XML Sitemaps • Manage Facets, Filters, & Search Pages w/ NoIndex • Handle parameters • Internal Link Optimization • Orphaned No More! • Model Crawled/Active Pages
- 22. Jori Ford | @chicagoseopro | #TechSEOBoost Expand Crawl Budget Containers & Clustering Services • Heroku Dynos • AWS EC2 • Docker & Kubernetes • Microsoft Azure Increase Google’s crawl by auto- scaling apps
- 23. Jori Ford | @chicagoseopro | #TechSEOBoost
- 24. Jori Ford | @chicagoseopro | #TechSEOBoost Remember… Google has limited resources There are only a few major factors you can manage: • Time to First Byte • Server Error Rate • Click-Thru Rate (CTR) • Freshness You can quantify your budget manually or with tools, but knowing it is critical Conserve. Optimize. Expand
- 25. Jori Ford | @chicagoseopro | #TechSEOBoost Thank You – @chicagoseopro /in/chicagoseopro www.joriford.com
- 26. Jori Ford | @chicagoseopro | #TechSEOBoost Thanks for Viewing the Slideshare! – Watch the Recording: https://youtube.com/session-example Or Contact us today to discover how Catalyst can deliver unparalleled SEO results for your business. https://www.catalystdigital.com/