1:07 AM

(3) Comments

About Robots.txt


Definition : It is a file that gives the permission to the search engines to control the the indexing of webpages.

Suppose you have few files or directories on the webserver which you want not to allow to the search engines to visit then you can control it by using Robots.txt file.

Search Engines uses spider to read Robots.txt file. To read it by Search Engines, save the Robots.txt file in the root directory. Suppose you have a website name http://www.yourname.com then your Robots.txt must be located here :

Syntax in Robots.txt : The two syntax used in Robots.txt :

User-agent : (Here user agent is spider eg. Google use googlebot as User Agent, MSN use msnbot as User Agent)
Disallow : (You can disallow particular directory or file to visit by spider)
Example :
User-agent : *
Disallow : /image.htm/
In this robots.txt file, you are allowing all the search engines to spider your website but you are not allowing to search engines to spider image.htm file of your website.

How to use it : To use it in the index page of your website, use it in META tag. The format is :
meta content="INDEX,FOLLOW" name="ROBOTS"
3 Responses to "About Robots.txt"
Smit said :
2:02 AM

Thanks for the robot.txt information. I really needed of it but now I can easily create it and hope you will help me also.
Hem said :
2:04 AM
Hi Smit,

Thanks for your visit. I will help you according to my knowledge. You can also take help from internet. Best of luck for your business if you are doing by you website.

D J said :
9:50 PM
Heyy Hem!

Another Intresting weblog! carry on!!!
And yes, your complain has been registered! Timely disposals of weblogs from me !! :)

Chill ..