Still Blocking Bad Bots With .htaccess?

I was searching for some ideas to write about for my Adult Blog Builder blog and came across an article that was written in 2006 titled "Block Bad robots, spiders, crawlers and harvesters". Depending on whether or not you have mod_rewrite you were given 2 examples of what to copy and paste into your htaccess file.

Given that this article is at least 5 years old, I was wondering if we should still be blocking those "bad bots"? And if so, where's a good place to pick up a recent list of bots to exclude?

I found this site that has an updated block list (Updated on: Sunday, January 15, 2012 EST):
http://www.bookofzeus.com/articles/b...s-and-spiders/ **

wow ... that's a lot of bad bots! I wonder if you can block bots like we do isp's on a server wide basis.

Have a look at CloudFlare. Recently Certified Hosting introduced it as an option on all shared hosting accounts. I was going to post my affiliate link but annoyingly I haven't made enough posts on the board yet grrrr :)

Quote:

---

Originally Posted by dirtyratstudios

Have a look at CloudFlare. Recently Certified Hosting introduced it as an option on all shared hosting accounts. I was going to post my affiliate link but annoyingly I haven't made enough posts on the board yet grrrr :)

---

I have upgraded your account :)!

Haha thanks Richard!

As I was saying recently Certified Hosting introduced it as an option on all shared hosting accounts.

CloudFlare blocks various threats including bots and the CDN side of things has certainly speeded up page response times. But I'm still testing it out really.

For anyone interested, CloudFlare is available for any website, and the basic features are FREE. And to be honest, I'm not seeing all that much more that you gain with the pro features. The basic is pretty loaded in and of itself. Give it a read.

The nice thing about it being integrated with the hosting is that you can enable it on any domain with one click.