PDA

View Full Version : zoom v5 and robots.txt


Bruce
07-19-2007, 07:13 PM
Hi I am trying to come up with a good robots.txt. This is my last try yet some URL,s were blocked that should not. Here is my robots.txt please for some help as to what I mite be doing wrong.

User-agent: *
Disallow:/

User-agent: BadBot
Disallow: /

User-agent: *
Allow:
User-agent: *
Allow:/support_center/
User-agent: *
Allow:/music/
User-agent: *
Allow:/searchworld/
User-agent: *
Allow:/forums/
User-agent: *
Allow:/band world/
User-agent: *
Allow:/band world/younggalaxy/

Ray
07-20-2007, 02:42 AM
I would suggest you refer to The Web Robots FAQ here, which contains a detailed explanation and many examples:
http://www.robotstxt.org/wc/faq.html

Not sure what you are trying to achieve with the above robots.txt, but it is in many ways flawed. For example, the very first two lines blocks off the entire site, so all other "Allow:"s that you apply afterwards are irrelevant. In most cases, you only need to use "Disallow:"s as all other paths that are found or requested by the robot is assumed to be allowed. It is also unnecessary to specify the same User-agent repeatedly.

Please see the FAQ above for more information.

Bruce
07-20-2007, 03:05 AM
thank you I found this one in the Google search. Liked it as the URL,s I want
to be indexed are the lesser of all URl,s.It seemed to be less work as there are
a lot of core URL,s to be denied!!!!:twisted: this robots.txt has already found the trash bin!:lol: