What Should be in robots.txt File?

[martindowling]

Hello

It has been suggested that I should have a robots.txt file on my site. If so are there any areas of Actinic that should be barred to spiders?

Best wishes
Martin

[RuralWeb]

I personally never use them.

[pinbrook]

I use robots.txt to disallow spiders from parts of a site I don't want to have indexed. In my case I disallow everything except the html pages

For an actinic site specifically some people disallow the cgi-bin as this was causing problems with google listings for people a while ago, you will need to search the forum to find these posts