When Google Comes Visiting

To be listed in Google’s database (or index), Google visits your site using automated
programs called robots or spiders. Such programs “read” each and every page of
your website, starting typically with your home page and then following each link to
all other web pages on your site. When a search engine robot or spider visits your
site, it is said to crawl or spider your site.

Important: Google will not add a web page to its index unless there is at least one
other web page in its index that links to one of your web pages. So don’t fret over
submitting your site to Google directly. Rather, you need to get another website to
link to your website first.

Website crawls are performed by the main Google spider, called Googlebot. Google
updates its main index regularly and now tends to go with a more continuous almost-
daily update although major monthly “updates” still happen. These major updates
usually correspond to major ranking algorithm changes (like the Florida Update). The
more “popular” your site, the more often it typically is crawled by Google. Highly
ranked sites and sites that update content frequently (like news sites) get crawled
daily.

The index update is generally based on content they have cached or stored in their
database previously. Because of the nature of the update process, ranking
calculations are performed multiple times for each page of every site. Because of the
sheer number of web pages in the Google index, these calculations can take a
awhile. Note that PageRank calculations still are done monthly (PageRank will be
discussed in detail later).

As such, search rankings can fluctuate - sometimes minute-by-minute. These
fluctuations were sometimes called the Google Dance as your site’s ranking can
jump around on a search results page (or even appear to drop completely). You can
view other versions of the index on different servers at the ten major Google
datacenters by using the Google Dance Tool at http://www.google-dance.com.



Important: It is critical that your website is up and running when Google visits you.
If your site is down, your listing on Google may disappear until the next update! The
reason is that Google thinks your site no longer exists and may remove it from the
index.

If interested, you can check your server log files for the user-agent “Googlebot”. This
will tell when Google crawls your site. You can also check by IP address although
this method is not as accurate as Google uses different IP addresses for their robots,

which can change over time. You can also look at the date displayed on the last line
of your website’s listing on a Google search results page.