Indexing Your Content is Optional (2020 Edition)
•
0 likes•148 views
Imagine having a big brand website with over 50% of its content not indexed by Google after weeks? Now imagine that the issue causing this severe decline in online visibility was self-induced? In the new digital marketing landscape, big brands losing money this way is shockingly normal. This deck from a Searchmetrics webinar Bartosz Goralewicz did with Marcus Tober on December 4, 2019. Want to watch the video? https://youtu.be/3XENVxVpyoo
Report
Share
Indexing Your Content is Optional (2020 Edition)
- 1. Bartosz Góralewicz linkedin.com/in/goralewicz @bart_goralewicz Indexing Your Content is Optional 2020 Edition
- 2. 63% of Walmart’s pages are not indexed in Google. H&M is losing 4,938,885 visits per month due to their indexing issues. 24% of Topshop products are not indexed in Google. Now that I have your full attention – let’s begin J
- 3. Back in 2016 I published Hulu’s case study… … that I wrote in SearchMetrics’ office in Berlin J one.ly/hulu2016
- 4. In 2016 JavaScript was… exotic?
- 5. Hulu vs. Netflix – the first JavaScript SEO battle
- 7. Fast forward to 2019
- 8. HTML HTMLHTML There is no such thing as an HTML website!
- 9. In 2019 * This means that their content isn’t visible with JavaScript switched off. ** this also means that Google needs to render the page to see that content. Rendering is optional though. of the websites use JavaScript for their crucial content.*80%
- 10. Brands with content relying on JavaScript
- 12. Let’s have a look at the indexing issue in the wild
- 13. case study
- 15. Let’s compare H&M’s crawl with and without JS rendering
- 16. Let’s compare H&M’s crawl with and without JS rendering
- 18. What usually relies on JavaScript? 01 Pagination 02 “You might be also interested in” (internal linking) 03 Top products 04 Reviews 05 Comments 06 Main content (rarely)
- 19. The Disconnect Google’s announcements this year didn’t help… Statement: Martin and John are nothing but helpful. Google is a large organization though and sometimes they make our life a little bit too complex.
- 20. But… but 90% of the websites get indexed in 5 seconds?
- 21. one.ly/martin5s
- 22. What is the exact timeframe for JavaScript indexing? Unfortunately - NO. 5 seconds for 90% of the pages?
- 23. Yeaaaah, my JS content can get indexed in 5 seconds. Well, not exactly.
- 28. Today’s episode was sponsored by the phrase: Queue latency
- 29. What Martin actually meant
- 30. Ranking
- 31. There is no timeframe for JavaScript indexing!
- 32. JS indexing delays are still an issue Good Bad
- 33. It is 2019 and our research shows that there are hundreds of thousands of domains not fully indexed. Even after… Months of publishing the content
- 35. Web development evolved. Most websites use ready-to-go components Slider Faceted navigation Related Products Comments Main content Menu The roots of the problem
- 36. Slider Faceted navigation Related Products Comments Main content Menu So when Google isn’t rendering your website, it may see…well, this:
- 37. 2 completely different websites and 2 different website graphs. Rendering delays lead to
- 38. Category Ladies/Men/DividedCategory Category H&M.comHOMEPAGE Sub-Category Sub-Category Sub-Category Jeans Products Products Products Products Shaping Skinny High Jeans JavaScript
- 39. Category Ladies/Men/DividedCategory Category H&M.comHOMEPAGE Sub-Category Sub-Category Sub-Category Jeans Products Products Products Products Shaping Skinny High Jeans JavaScript
- 40. Let’s compare H&M’s crawl with and without JS rendering
- 41. Let’s compare H&M’s crawl with and without JS rendering
- 43. Rendering is not Google’s indexer default mode. Rendering is triggered by... Heuristics
- 44. Rendering heuristics So, you look at the difference between the initial HTML, and, then, if after rendering you see extra content? Yeah.
- 45. There are certain heuristics, that, if we see after a while, like, oh, this page, actually, the renderer does not diff as much or doesn’t diff, it looks the way before The Heuristics I still haven’t fully, like, grasped what exactly triggers the heuristics
- 46. are FAR from perfect though. …those heuristics
- 47. Every website, when we see them for the first time, goes to rendering. ALL NEW SITES GET RENDERED
- 48. But what is a new website exactly? What is a new website?
- 49. Rendering Heuristics Logic behind those heuristics Limitations Changes in how we should approach web development • Staging • Experiments
- 50. Rendering vs. Crawler budget
- 51. Rendering vs. CPU cost
- 53. WRONG
- 55. CPU is not affecting rendering* *within what we would call a “reasonable” load
- 58. Is the JavaScript rendering cost affecting crawling and indexing? It’s complicated
- 59. How to check the rendering cost of a page TL;DR – Too Long Didn’t Render onely.com/tools
- 60. TL;DR – Too Long; Didn’t Render onely.com/tools
- 61. onely.com/tools
- 62. onely.com/tools
- 64. Percentage of JavaScript content indexed after 14 days 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 100% 96% 70% 34% 0,5% New York Post Eventbrite Target The Guardian CNBC
- 65. TGIF - The Google Indexing Forecast onely.com/tools/tgif
- 66. HTML vs. JavaScript indexing
- 67. HTML vs. JavaScript indexing
- 69. WWJD - What Would JavaScript Do
- 70. WWJD – BBC & meta tags vs. rendering
- 71. WWJD – BBC & meta tags vs. rendering
- 72. WWJD – BBC & meta tags vs. rendering
- 73. WWJD – BBC & meta tags vs. rendering
- 74. WWJD – Links added by JS
- 75. WWJD – Links removed by JS
- 77. Even without JavaScript dependencies big players struggle with indexing 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Indexed after 1 day Indexed after 2 days Indexed after 1 week Indexed after 2 weeks The Guardian Target Reuters Eventbrite 98.5% 3.5% 68.5% 32% 99.9% 33% 88% 47% 99.9% 78% 91% 57% 99.9% 84% 91% 58%
- 78. Wrapping up
- 79. Parts of your website depend on JS 01 Webmaster updates with new products 02 Google crawls your website without seeing all the links 03 Only crawls a part of the domain without finding products (valuable content) 05 Budget is too low to render JS 04 Google indexer is confused - crawler budget falls
- 80. Which leads to this… 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Indexed after 1 day Indexed after 2 days Indexed after 1 week Indexed after 2 weeks The Guardian Target Reuters Eventbrite 98.5% 3.5% 68.5% 32% 99.9% 33% 88% 47% 99.9% 78% 91% 57% 99.9% 84% 91% 58%
- 81. TO DO: Go to onely.com/tools and check your website (100% free) Crawl your website with and without JS and compare two crawls
- 82. We are going like: ”Oh, all right, we are gonna skip rendering.” It is not as frequently happening anymore. (...) Many websites even if they do not run JavaScript, they might still go through the render phase, because it doesn’t make a difference as much Crawling and indexing to come together I expect, eventually rendering, crawling and indexing will come closer together
- 83. Summary Indexing your content is one of the key challenges for 2020 All indexing issues we are seeing are 100% self-induced by website owners We need to redefine how we are looking at the crawler budget and the factors affecting it JavaScript SEO = Technical SEO
- 84. To do: 1Closely monitor your indexing stats (daily, weekly, biweekly). 2 If your content relies on JavaScript, monitor it closely and make sure that: • The code is light and efficient (cheap to render) • Your code doesn’t affect web performance on low-end devices 3 Make sure that no content within your structure relies on JavaScript (check all the page types within your domain). 4 Make sure that no meta data is changed during rendering.
- 85. To do: 1Closely monitor your indexing stats (daily, weekly, biweekly). 2 If your content relies on JavaScript, monitor it closely and make sure that: • The code is light and efficient (cheap to render) • Your code doesn’t affect web performance on low-end devices 3 Make sure that no content within your structure relies on JavaScript (check all the page types within your domain). 4 Make sure that no meta data is changed during rendering. Subscribe to Onely.com – be the first one to know about our crawl budget monitoring toolset.
- 86. More data is coming soon
- 87. Cloaking in 2019 bartosz@onely.com linkedin.com/in/goralewicz @bart_goralewicz THANK YOU one.ly/15min