Securing Your Adult Site - Inside The Spider Hole

Greetings All:

This tutorial will help teach you an easy and cost effective (read: free) way to all but put an end to those nasty nasty little offline downloaders once and for all.

Before reading any further, please read Step 6 at the bottom. I mean it, I'm asking nicely, please? It's for your own good, honest it is.

By the time you are reading this paragraph, you should already have read Step 6 at the bottom. If you haven't, you don't listen and will fuck the rest of this up anyway, so you might as well stop reading now.

Step 0 - Background Information:

Some people that visit your website are evil. They will download and run one of many popular applications which exist for the sole purpose of downloading the entire contents of your website to their computer for later viewing, or to use on sites of their own. This type of activity is very costly to you as a webmaster, as it adds additional overhead to your server (slowing it down for everyone else), and additional monetary expense (all that bandwidth).

The most common way that people attempt to rid their sites of these nasty little creatures, is to develop long lists of "user agents" (a way that applications use to identify themselves to your webserver), and use mod_rewrite (an apache module used to redirect requests) to throw them to the wind. Unfortunately, this methodology is completely ineffective for many reasons:

1. The list is constantly needing to be updated as new downloaders and developed and released.
2. Many of the most popular downloaders spoof themselves to look like a legitimate browser, such as Internet Explorer.
3. These vast lists add significant overhead to your server.
4. They don't prevent the user from switching to a different downloader, if the one they were using fails due to your list.

So, what is one to do? We're going to do behavioral profiling to find these things and stop them dead in their tracks! Sounds impressive and frustratingly complicated doesn't it? Keep reading....

These programs work by "spidering" your website, producing a list of all of the links from one of your pages to another, visiting each of them in turn, and downloading all of the images and movies they find. Note: These programs visit ALL of the links that they find in the html returned to it when it requests a page from your server. This is where their behavior is different than the behavior of a real person. People only visit links (aka click on them), that they actually see. Muahaha!

So, here's what we're going to do. We're going to add some links on your website that your users won't be able to see, but the spiders will. When the spiders visit these links, BAM our trap is sprung! We will then ban not the program, but the evil nasty nasty user that decided he was going to be an SOB and run it against your site in the first place!

Step 1 - Create The Trap

Our secret trap, if you remember, is simply a link that the spider can see but your users can not. Here's what you'll be adding to some of your pages:


.


Ok, this looks way more complicated than it is. The begin and end somsbitch lines are simply comments. These will not be visible on your website, and are just used to denote where our trap begins and ends. You can remove them if you want. Next, we have our link. The only thing you really have to do here, is make sure that font color="#" is the same color as the background of your webpage. This is done in Hex, so look it up if you are unsure what the hell those six numbers are (#000000 is black #ffffff is white). The onmouseover will give your user a warning message should they miraculously find that tiny little invisible period that we're using as a link. The onclick will stop them from visiting the link if they're using any browser that has javascript enabled, and they're really stupid and choose to ignore our warning. Remember, we are not kidding!

Just copy and paste the secret trap to several of your pages. Your first members index page, and the first couple of gallery pages that you have will be more than enough. It's best to add the link at the very top of each page, so that the spider will spring the trap early on!

(Side Note: There are a small number, maybe 1 or 2, downloaders that will look for link text that is the same color as page background. If you're a real pro, you can create a small image that's the same color as your page background, and use it instead of the text link I mention above)

Step 2 - Create The Shit List

Yes, we are going to put any user that tried to download your entire site to their computer onto a Shit List! Now you need to create a blank text document called "ShitList.txt" and upload it to the root public directory of your webserver. This folder is usually called "www" or "public_html" or "htdocs". Now, you need to make ShitList.txt writable. From the command line you can do a chmod 666 ShitList.txt or if you're using an ftp program you can set the file writable from there (check the help file of the particular ftp program you are using for information on how to do this).

Step 3 - The Magic VooDoo File

For the sake of this tutorial, I'm not going to sidetrack and teach you how to program in perl. So, I have attached a perl file as a zip to the end of it. Download it onto your computer and unzip it. Open it up, and change the "/path/to/your/ShitList.txt". It's clearly marked, and all you have to do is provide the full path to the ShitList.txt file that you just created in step 2. Now, search for YOURWEBMASTEREMAIL@ADDRESSHERE.COM and replace it with, you guessed it, your webmaster e-mail address. Upload this file to your webserver's "cgi-bin" directory.

Step 4 - Put The Trap Into Action

It's all there. The secret link that's bait for the nasty nasty downloader, the Shit List, and our Magic VooDoo File. Now all we have to do, is set the trap!

First, change /path/to/your/ShitList.txt below to the actual path to where your Shit List is. This will be the same path that you added to the Magic VooDoo File in step 3. Then, copy and paste the updated rewrite rules into the httpd.conf file that's located in your apache config directory. Totally confused by this step? Good, this is a perfect opportunity for you to take advantage of the "managed" portion of the "managed hosting" plan that you currently have. Email the techie at your ISP and have them add it for you.

RewriteMap ShitList txt:/path/to/your/ShitList.txt
RewriteCond ${ShitList:%{REMOTE_ADDR}|NOT-FOUND} !=NOT-FOUND
RewriteRule .* - [F,L]
RewriteRule ^/somsbitch\.html$ /cgi-bin/ShitList.pl [L,T=application/x-httpd-cgi]

Step 5 -
Wait, There Is No Step 5 We're Finished! W00t!

That's it. When one of your users tries to use a downloader, it will spring our trap, and add the user's IP address to your Shit List. (begin run on sentence) The spider will quickly give up, and when your user tries to visit your site, they'll see a friendly little message telling them to e-mail you so that you can tell them what a piece of crap they are and that if they ever try that stuff with you again you're going to delete their account and keep their subscription fee! (end run on sentence) After you're done bitching at them, you can delete their entry in your Shit List.

Keep in mind, most users have dynamic IP addresses, so it's likely they won't be banned forever if they spring your trap. However, they will be stopped cold until their IP changes (which can take hours to days or even weeks depending on the ISP), they will see the warning message, and they will have to contact you if they want their access back right away.

Clean out your Shit List once a week or so (if you want to be fancy you can do this automatically with Cron, but that's another tutorial) to make sure that you're not accidentally banning innocent users, and life will be good.

Step 6 - Words of Caution!
Wait There Wasn't Even Supposed To Be A Step 5! What The Hell!

First, this tutorial is for the Apache Webserver that has perl cgi access. If this doesn't apply to you, sorry you are shit out of luck.

Second, DO NOT UNDER ANY CIRCUMSTANCES never EVER add this trap to an area of your site that isn't password protected. This is for member's areas ONLY! Do NOT add it anywhere else. Do NOT, Do NOT Do NOT NOT NOT NOT NOT.

If you do, you will spring your trap on every good natured spider that visits your site, this includes thing like, oh, say, hrm, GOOGLE. If you put this anywhere but inside your protected members area you will ban google and every other search engine from visiting your site, you will no longer be indexed in any search engine, you won't get traffic anymore, and you will go out of business. Don't e-mail me asking WTF, and no you can not come live with me if you do, because I warned you!

Did I mention that you should NOT add this trap to anywhere but INSIDE YOUR PROTECTED MEMBER'S AREA? I did? Ok, good.

Step 7 - What The Hell! Is This Stupid Tutorial Every Going to End?!?!

1. I take NO responsiblity if you manage to delete your entire life's work while trying to follow my tutorial. Follow my directions, and use the Magic VooDoo File at your own risk!
2. My tutorial was inspired by a long winded technical rant written in 2001 by a Mr. Killough.
3. Yes, I realize that my tutorial turned into a long winded rant too, but at least it's not technical, and it was written in 2005 by me, and I'm way cooler than Mr. Killough.

Bump for you and that's a great tutorial. As an original script lover I will take this tidbit and forever enshrine it into my personal archive of cool scripts.

I use 2 scripts coded to work with one another. My member php script and a script that re-writes basic authentication to control access and uses two environmental variables plus cookies to determine whether or not multiple users using the same username are in my members area.

It can suspend them, but them on probation, expel them and/or sent them an e-mail and lock them out for 6 hours if they are "greedy".

Basically, a user in a brower could not launch a downloader, but .. they cost money and in the sense of security, I'll add this cool trap as well :D

I like it a LOT.

If you want to you may include it on my site with a link back to you as a tutorial. I'd like to list it in the tutorial links on the left. http://adult-site-traffic.com/adult_...c/linkme.shtml .

:)

AST

.