Crawl optimization settings - Yoast SEO Features • Yoast

Make your site green and search engines happy

We’re on a mission to reduce website crawl load — making it a more straightforward process for crawlers. As a result, we’re reducing the strain on your website and the environment. With the crawl settings in Yoast SEO, these two things come together. By selecting which resources and URLs are automatically added by WordPress to discard, you make your site easier to crawl and increase the chance that Google will crawl the URLs you want.

Control how search engines crawl your site

Makes your site more sustainable
Block crawling useless URLs and assets
Prevent spam attacks on your site

Why do you need the crawl optimization settings in Yoast SEO?

Getting every possible URL on your site crawled, indexed, and ranked by a search engine is nearly impossible. Every crawler has a limit to what it can do at any given time, so it’s best to focus on getting your most important URLs crawled. Unfortunately, a CMS like WordPress comes with a lot of baggage: URLs that it automatically adds to your site, unnecessary scripts, useless metadata, and much more. What if you could throw this all out?

Unlock crawl settings in Yoast SEO Premium

Get this feature and much more for your WordPress site with the Yoast SEO Premium plugin!

Get Yoast SEO Premium Only $99 USD / year (ex VAT)

Making it easier to get your essential URLs crawled

Crawling processes are very resource-intensive, and with a never-ending stream of new content to discover, search engines like Google have to be selective in what they can do. This means that search engines cannot come by all the time. So if they come, you better offer the most important URLs to them — the ones you want them to crawl! The crawl settings let you turn off certain types of URLs, scripts, and metadata that WordPress automatically adds to enhance crawling.

Select which WordPress feeds you want to keep or discard

Making your site more sustainable

Crawling costs search engines a tremendous amount of energy. Search engines like Google operate tens of data centers, helping them to crawl, index, and rank information from around the world. All of these use incredible power to function, but the same goes for your site. The server you host your site on uses power, and every request to your site costs energy. So, to make the whole process more power-efficient, use the crawl settings to turn off stuff that has no reason to be crawled. Together, we can make your site — and the web — a lot greener!

Crawl settings help clean up your site

You can see the crawl settings as a way to easily clean up your site of unwanted overhead. For instance, you’ll also find options to clean up the internal site search of your site to prevent SEO spam attacks. And a way to clean up your permalinks by filtering campaign tracking and unregistered URL parameters, which leads to fewer duplicates found by search engines.