I was searching for some ideas to write about for my Adult Blog Builder blog and came across an article that was written in 2006 titled "Block Bad robots, spiders, crawlers and harvesters". Depending on whether or not you have mod_rewrite you were given 2 examples of what to copy and paste into your htaccess file.
Given that this article is at least 5 years old, I was wondering if we should still be blocking those "bad bots"? And if so, where's a good place to pick up a recent list of bots to exclude?
Bookmarks