Results 1 to 3 of 3

Thread: Robots.txt & .htaccess

  1. 01-19-2006, 03:55 AM #1
    DEVELISH
    DEVELISH is offline
    Life is a dick and when itīs get hard---just fuck it... DEVELISH's Avatar
    Join Date
    Jul 2005
    Posts
    2,367

    Robots.txt & .htaccess

    Hi,

    over in a German forum (http://www.pornhulk.net/) we plan to share our robots.txt and .htaccess lists for blocking IPs and Crawlers/Spiders.

    Would anybody of you mind to contribute?

    DEVELISH
    Reply With Quote Reply With Quote

  2. 01-19-2006, 05:09 AM #2
    Lee
    Lee is offline
    You do realize by 'gay' I mean a man who has sex with other men?
    Join Date
    Oct 2003
    Location
    New Orleans, Louisiana.
    Posts
    21,635
    I beleive there is an article in that area of the community forums about blocking leeching robots and applications, feel free to share the list of referral agents contained within it

    Regards,

    Lee
    Reply With Quote Reply With Quote

  3. 01-19-2006, 09:31 AM #3
    DEVELISH
    DEVELISH is offline
    Life is a dick and when itīs get hard---just fuck it... DEVELISH's Avatar
    Join Date
    Jul 2005
    Posts
    2,367
    Thanks Lee,

    I will share the UAs and IPs with y'all.

    Over at pornhulk we are talking about building a shared database.

    PLEASE contribute your robots.txt and .htaccess block-lists

    DEVELISH
    Reply With Quote Reply With Quote

« Previous Thread | Next Thread »

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Forum Rules