One of the first questions that new website and blog owners ask is: How often is my site indexed? Answering this question can be a challenge simply because indexing is probably not the best word to use in this case. Rather, the question, “how often is my site crawled?” may be a better way to put it. Let’s take a look at the difference between the two so you can better understand how the process works.
Indexing and Crawling – What’s the Difference?
Search engines have what they call an “index.” This is, essentially, a collection of all the web pages they deem worthy of being listed in their search results. When a new website or blog is published on the web, it is not automatically added to any search engine index. The search engine’s spiders must first crawl your web pages to gather data about what your site is about. Once the spiders crawl the pages of your site, the search engines will start indexing relevant pages gradually.
Crawling is a bit different. This is what search engine spiders do when they land on your website. They crawl through each page and collect information about the page and its topic.
How Often Are Sites Indexed?
Being indexed is kind of like being added to the Yellow Pages. It’s a one-time deal. Unless you make revisions to the page, there’s really no reason to index it again. If you add more pages to your website, those will be indexed as well providing they are deemed worthy of being a part of the search engine’s index. To put it simply, your site will only be indexed once if you never add any new pages.
How Often Are Sites Crawled?
Most websites are crawled at least once per week. Sites that are frequently updated and use a pinging service tend to get crawled more often simply because their site is so active. Static web pages may be crawled less often as there is no new information to be added to the results.
A website’s homepage may be indexed after only a few days, or it may take a few weeks depending on the nature of the website and the quality of its content. If a web page has a few strong backlinks pointing to it, its pages will likely be indexed quicker than a page that has low quality links or no links at all. Some websites are never indexed, or are removed if deemed irrelevant or provide little value to the visitor. A “404” error page is a great example of this. Google, for example, may index this page for a short time and then remove it after determining the page provides no value to a visitor.