Printable View
I would like to block some duplicate pages that my script is producing via robots.txt.
I want to block this page: http://www.example.com/cgi-bin/pseek...tegory_widgets
Would this work to block the url from being indexed by search engines?
User-Agent: *
Disallow: /cgi-bin/pseek/dirs.cgilv
Or would it be better to write out the full URL for each page I want to block like this.
User-Agent: *
Disallow: /cgi-bin/pseek/dirs.cgilv=2&ct=category_widgets
PowerSeek is acually a funky script, even if you do block it in a robots.txt file, you are still going to have listings in the SE's for anything running out of your cgi-bin directory because of how the script itself works.Quote:
Originally Posted by Northstar
A better option, IMHO, would be to use the static HTML page generation instead of the dynamic version of PSeek that way, you shouldnt have as many problems with the cgi related script calls finding their way in to the search engines :)
Hope this helps.
Regards,
Lee
Thanks for the advice. You are right PowerSeek is a good script but it has a lot of quarks. I have been using this since the release a couple years back. I never even noticed the duplicate pages till recently. I would like to change over to static but I have good positions in Google and Yahoo with the dynamic URLs and if I change the URLs I could loose those positions for awhile. I'm going to try to use this robots.txt file and hope it blocks all the duplicates. I just hope it doesn't inadvertently block dirs2.cgi or dirs.cgi?st also. From all I read I think this robots will work to block the content I want blocked.
# All robots will spider the domain
User-agent: *
Disallow: /cgi-bin/pseek/dirs.cgi?lv=1
Disallow: /cgi-bin/pseek/dirs.cgi?lv=2
Disallow: /cgi-bin/pseek/dirs.cgi?lv=3
Disallow: /cgi-bin/pseek/modify.cgi
Disallow: /cgi-bin/pseek/rprtb.cgi
Disallow: /cgi-bin/pseek/addurl.cgi
Disallow: /html/