Design Optimization- The Power of robot.txt File
Utilizing Robots Exclusion Protocol, robots.txt has the ability to instruct the Search Engines not to index certain part of your website. This file must reside in the server's root directory. In other words, put this file in the same place that you put your home page. A typical URL for robots.txt is as shown:
www.useseo.com/robots.txt
What about Meta Tag Robots Exclusion?
Most Search Engines do not honor this type of Meta Tags. Therefore, it is a waste of time using these tags to instruct the bots to index or not index certain page. The following shows how to use these tags for exclusion (it is advisable to use robots.txt instead):
<HEAD>
<TITLE>Page Title Here</TITLE>
<meta name="robots" content="noindex,nofollow">
</HEAD>
How robots.txt works
The following codes tell the bots not to index any part of your website:
User-agent: *
Disallow: /
User-agent enables you to specify which bots should follow the directions on the second line. An
asterisk (*) means 'all robots'.
The Disallow command allows you to specify the directories or filenames that you do not want
the Search Engines to index. In the above example, a forward slash (/) instructs the Search Engine
to ignore everything in your root directory. Therefore, Search Engine will not index all the pages
in the website.
The following example instructs Named Bot not to index contact.html:
User-agent: Named Bot
Disallow: /contact.html
The following example instructs all Search Engines not to index the partners directory:
User-agent: *
Disallow: /partners/