About Robots.txt

Definition : It is a file that gives the permission to the search engines to control the the indexing of webpages.

Suppose you have few files or directories on the webserver which you want not to allow to the search engines to visit then you can control it by using Robots.txt file.

Search Engines uses spider to read Robots.txt file. To read it by Search Engines, save the Robots.txt file in the root directory. Suppose you have a website name http://www.yourname.com then your Robots.txt must be located here :
http://www.yourname.com/robots.txt.

Syntax in Robots.txt : The two syntax used in Robots.txt :

User-agent : (Here user agent is spider eg. Google use googlebot as User Agent, MSN use msnbot as User Agent)
Disallow : (You can disallow particular directory or file to visit by spider)
Example :
User-agent : *
Disallow : /image.htm/
In this robots.txt file, you are allowing all the search engines to spider your website but you are not allowing to search engines to spider image.htm file of your website.

How to use it : To use it in the index page of your website, use it in META tag. The format is :
meta content="INDEX,FOLLOW" name="ROBOTS"