It all takes time
There's more than just Google
Check your robots.txt file
How do I submit sites to Google?
Be prepared for uncertainty
Check your log files
Use 'Google machines'

It all takes time

As a rough estimate it takes the average new site at least a month to have all its pages crawled and indexed in Google, usually two. That can be a long time while you wait impatiently for the first visitors to arrive via your site's listings in a search engine. The best course of action is to use that time to write more content, find reciprocal links or something completely different. Have patience and forget about Google for a while!

Top

There's more than just Google

While Google dominates the search engines at the moment, there's other places to promote your site for free. DMOZ, also known as the Open Directory Project is worth submitting to although it can be hard to get listed in some categories. AllTheWeb accepts free submissions, as does MSN and Altavista.

Top

Check your robots.txt file

Make sure your robots.txt file doesn't have any entries that can prevent your site being indexed. In case you haven't heard of them, a robots.txt file is mostly used to tell a complying search engine spider to keep away from certain files or directories. Valid entries look something like this (this would tell all spiders to keep away from the /forums and /stats directories):

User-agent: *

Disallow: /forums

Disallow: /stats

More information on writing a robots.txt file can be found at the Web Robots Pages. You can also use META tags to control spiders. Check the META standards for spiders information at W3C.

Top

How do I submit sites to Google?

You can submit a site to Google, but in most cases you won't need to. Google's spiders will follow inbound links to your site and find it for themselves. If your site doesn't have any inbound links then you should submit it, although without them it will probably not be a very successful site. Try to get at least one inbound link - a local or country specific business directory is often a good place to start.

Top

Be prepared for uncertainty

Due to the size and scope of the Google index, it is reasonable to expect that results may vary from time to time. A vast number of computers are used to serve Google's search results and the search engine rankings for a site can change depending on the datacenter that was accessed for the query.

One day your site may be in the top 10 for a key phrase, the next day it may be lost on the 10th page. There will always be fluctuations in the rankings, and the best defence is to create many pages of different content so that no one key phrase drives all the search engine traffic to your site.

Top

Check your log files

Your server log files can tell you when a Google spider hits your site. This can be useful if you are waiting for the first visit from Google's spiders (once your site is being regularly spidered you will see multiple entries in your log files). The 'User-Agent' string you should look for in your log files is something similar to this:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

By studying your log files you can check which pages are being picked up by the spider and identify any problems if they arise.

Top

Use 'Google machines'

There are 'Google machine' web sites where you can check a search query at multiple Google datacenters. These are for the truly Google obsessed! Now that the Google update is more unpredictable they are less useful, but they can still tell you some interesting things about the status of your site.