PDA

View Full Version : Spiders and Site Map

Steven Barkess
03-Apr-2005, 02:48 PM
Hi

I may be way off here, but at the back of my mind I am sure someone said to me that spiders like real links (ie www.site.co.uk/section1), and the links in my site between sections are not true links as they call from the cgi-bin (ie www.site.co.uk/cgi-bin/section1).

Looking at this more, I note the site map works on the latter too and use the cgi-bin.

So, is there any truth in this.

I am trying to increase the chances of any spiders coming to my site not hitting dead ends so have created true links to about 8 sections on my site on each page. I am wondering tho if this is necessary.

Thanks for any thoughts.

Steven
Stunty
03-Apr-2005, 06:39 PM
Are you using frames? - If so, then I think some of the spiders may have a bit of difficulty negotiating around the brochure pages. Apart from that, you don't have anything to worry about - I used to have the same concerns as yourself. however, since then, I've found that google, (and pretty much all the others), find their way round the catalog, and out the other side OK.

I will admit, I am at a loss as to how Actinic achieves this with the cgi's -
maybe somebody around here would be kind enough to explain it to us??
:rolleyes:
pinbrook
03-Apr-2005, 07:03 PM
sitemap has spiderable links.
Stunty
03-Apr-2005, 08:59 PM
Ahhhh..... I see...

I presume this line: '<LINK REL=STYLESHEET HREF="actinic.css">' is the fella responsible then huh?
pinbrook
03-Apr-2005, 09:23 PM
<LINK REL=STYLESHEET HREF="actinic.css">

Ug!! no, thats the style sheet ie how to make the site pretty
Stunty
04-Apr-2005, 05:22 AM
OK, I've been studying the source code from my site map on & off now for a good four hours - And, well, it's driving me crazy!! I can not work out which part of the code is responsible for linking up the spiders!!

please, somebody, put me out of my misery???

LOL
pinbrook
04-Apr-2005, 11:21 AM
Taken from http://newtrecords.co.uk/acatalog/__webcat-a2d.html the links on "GO" are not spiderable http://newtrecords.co.uk/cgi-bin/ss000001.pl?PRODREF=tg260lp&NOLOGIN=1

Taken from http://www.newtrecords.co.uk/acatalog/index.html

this is spiderable

http://www.newtrecords.co.uk/acatalog/404_Records.html
Stunty
04-Apr-2005, 12:35 PM
Yep! - I get it now. Anything to-&-from a straight '.html' link can be traversed by the spider. Yet anything which calls upon a script to find its destination isn't spiderable....

Cool!

Thanks for that. :D