Robots and spiders

[Steven Barkess]

Hi

Taken from http://www.seekport.co.uk/seekbot/ is it necessary for the following, I thought not.

Robots

<meta NAME="robots" CONTENT="index,follow">
<meta NAME="robots" CONTENT="noindex,nofollow">

The crawler reads this information to discover if the author has given permission to crawl the page.

* (CONTENT="index") to give permission
* or (CONTENT="noindex") to deny permission.

[wjcampbe]

Robots in text file or META format is only needed to try to stop a spider from going somewhere on your website. (I say try because several spiders ignore the robots file).For example a test folder. Follow and index is the normal behaviour of spiders.

W3Schools have some good information on this.