How Google Discovers, Crawls and Serves Web Pages

How Google CrawlsUnderstanding the basics of how Google discovers, crawls and serves up web pages, is an important key to getting a website, and it’s pages, recognized in the search engine.

Everyone is wanting their website to rank high in search results, so you are competing against thousands, and even millions of other websites. What is the process Google goes through to discover a website and serve it up in a web search?

To help you understand this process, consider that when you are searching the web you are opening up a very large book,  a huge book, with an impressive index outlining where everything is located. When a search is performed, Google’s program (Algorithm) checks their index of millions of pages and serves up the most relevant results in a ranking fashion. The processes Google uses for delivering the most relevant results is by crawling, indexing and serving.

From Google’s Webmaster Tools help section they explain each of these processes:


Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.

We use a huge set of computers to fetch (or “crawl”) billions of pages on the web. The program that does the fetching is called Googlebot (also known as a robot, bot, or spider). Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

Google’s crawl process begins with a list of web page URLs, generated from previous crawl processes, and augmented with Sitemap data provided by webmasters. As Googlebot visits each of these websites it detects links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

Google doesn’t accept payment to crawl a site more frequently, and we keep the search side of our business separate from our revenue-generating AdWords service.


Googlebot processes each of the pages it crawls in order to compile a massive index of all the words it sees and their location on each page. In addition, we process information included in key content tags and attributes, such as Title tags and ALT attributes. Googlebot can process many, but not all, content types. For example, we cannot process the content of some rich media files or dynamic pages.

Serving results

When a user enters a query, our machines search the index for matching pages and return the results we believe are the most relevant to the user. Relevancy is determined by over 200 factors, one of which is the PageRank for a given page. PageRank is the measure of the importance of a page based on the incoming links from other pages. In simple terms, each link to a page on your site from another site adds to your site’s PageRank. Not all links are equal: Google works hard to improve the user experience by identifying spam links and other practices that negatively impact search results. The best types of links are those that are given based on the quality of your content.

In order for your site to rank well in search results pages, it’s important to make sure that Google can crawl and index your site correctly. Our Webmaster Guidelines outline some best practices that can help you avoid common pitfalls and improve your site’s ranking.

Google’s “Do you Mean” and “Google Autocomplete” features are designed to help users save time by displaying related terms, common misspellings, and popular queries. Like our search results, the keywords used by these features are automatically generated by our web crawlers and search algorithms. We display these predictions only when we think they might save the user time. If a site ranks well for a keyword, it’s because we’ve algorithmically determined that its content is more relevant to the user’s query.

Naturally, it is important to have Google crawl and index as many relevant website pages as possible. Understanding the processes that Google uses to produce the most optimal website pages helps to get your web pages crawled, indexed and served up with the highest ranking possible.

Leave a Reply

Your email address will not be published. Required fields are marked *