Quote Originally Posted by gaydemon View Post
Duplicate content can be avoided by simply having a extra line in your robot.txt file which tells google etc not to index a part of your website - for example the part which you are working on. I often build pages on my sever which i simply exclude from getting indexed using robot.txt file, when its ready to be listed on google i again allow it to be indexed.
Bjorn is right and I never even thought about the duplicate content issue. This robots.txt thing is a nice idea and should work in theory. But I have a domain where I've robots.txt'd some pages and told Google not to index them because they're duplicate pages, but it looks like I'm still getting dinged for having duplicate content. So I'm rather dubious about how effective robots.txt is.

Michael