Anyone Know Which SE's Still Follow The Robots Protocol?

[Lee]

Im hearing contradicting information about this.

Some are saying that Google does follow rules in a robots.txt file whilst others are saying that it doesnt

Anyone know which school of thought it correct?

Regards,

Lee

[gaystoryman]

I'd say the big boys do, Y, MSN, G... in checking my logs I have some folders blocked, and they haven't crawled those pages... now it could be they haven't found it though unlikely that they have found other pages put up way afterwards... so I'd say they do. It is an easy file to put up and while the nasty one's don't follow it, still, imho, a valid file to have.

just my opinion
Ian

[Dzinerbear]

I've never had a problem with Google and my robots.txt I have never found any of my "hidden URL" galleries in Google.

dzinerbear