What is Crawling and Why Does Google Do It?

There are estimated to be 1.5 billion active websites as of 2022, to put that into perspective with the approximate global population being at just under 8 billion, roughly every 5th person has their own website. The size of the internet is nothing short of colossal and in the seconds you’ve taken to wrap your head around just how crazy these stats are, several new websites have popped up at the rate of 2 or 3 a second. Discovering new websites is Google’s role as the leading search engine and it carries out this process through a piece of software that is referred to as a crawler. What is a Crawler? A crawler is a piece of software that looks to visit as many websites as possible discovering new ones and reporting back information on those that it visits. The information it collects is then used to create an index of websites on the internet allowing for easy retrieval of data and access for users. Crawlers can also be referred to as spiders, ants or indexers. Google uses this information to gauge the overall quality of the website in turn boosting or lowering its SEO position. The key factors that influence this decision come under the ‘core web vitals’ measurement criteria which encompass; LCP (Largest Contentful Paint), FID (First Input Delay), and CLS (Cumulative Layout Shifts). To find out more about what each of these measurements cover, visit our core web vitals blog. The Two Types of Crawlers While the function of a crawler is pretty straightforward, the work they do is anything but. Even though the complexity of their task, there are only actually two different types of web crawlers. The ‘discovery’ web crawlers’ function is to find new pages on your website, it does this by following links – usually from your home page- to different sections of the site looking for any changes that may have been made since the last crawl was carried out. You also have the ‘refresh’ crawler which revisits pages that have already been crawled to take new data readings and update the information that Google has on the overall site. Why Does Google Do This? Google isn’t the only search engine that makes use of crawlers. Bing, Baidu, Duck Duck Go and all the other search engines you’re likely to ever use have their own versions of the common web crawler which generally do roughly the same thing. The information that the crawler collects is key to the search engine, this information determines the overall quality of the website and plays a huge part in SEO rankings. Through the use of crawlers, Google can quickly identify harmful or spammy websites making them more difficult to find for users while also boosting the reach of websites that do meet the criteria.

For more information on What is Crawling and Why Does Google Do It? talk to Odyssey New Media Ltd