Did you make changes to your website recently but are not seeing the new web pages in Google, Bing or Yahoo? That’s because your website changes have not been ‘crawled’ as of yet. If you want a better explanation of what a web crawler is, please visit “Web Crawler” at Wikipedia.com.
Search engines can only crawl so many websites at a time, so your web site may be deep-crawled only once a month, although it may skim the surface of your website every other day looking for changes. This is why Search Engine Optimization can take some time and effort and is not a quick task. Good website SEO practices are rarely ever “finished” as it is a continuous fact of website life. I found this quote online which sums it up nicely;
Whenever somebody asks me how long search engine optimization takes, I counter by asking them, “How long does marketing take?” – AustinSEOguy.com
So how do you tell when your website was last crawled? Go to Google or Yahoo, search for your website. Next to your website’s listing in the organic search results, you will see the word “cached’ next to it.
Once you click the cached link, it will bring up the version of the website when it was crawled or cached. At the top of the cached version, which will have some design flaws usually, there will be a line of text which should state something like;
This is Google’s cache of http://www.ocalawebsitedesigns.com/. It is a snapshot of the page as it appeared on Feb 7, 2011 21:27:24 GMT. The current page could have changed in the meantime. These search terms are highlighted: ocala website designs
So how do you tell the search engines to re-crawl your website? You really can’t. All you can do is re-submit the webiste back into the queue and wait. Here are the direct links to re-submit your site to the three major search engines:
According to Dummies, “Google crawls the Web at varying depths and on more than one schedule. The so-called deep crawl occurs roughly once a month. This extensive reconnaissance of Web content requires more than a week to complete and an undisclosed length of time after completion to build the results into the index. For this reason, it can take up to six weeks for a new page to appear in Google. Brand new sites at new domain addresses that have never been crawled before might not even be indexed at first.”
Read more at: Dummies.com – Timing Google’s Crawl