Printable View
Ive been going through the server log files on GWW and am seeing the same ip addresses coming up time and time again hitting pages that, quite frankly, 'real' people shouldnt be hitting lol
If anyone has a list of the search engine spiders ip addresses handy it would be a great help if you could share them so i can narrow down whats happening :)
Regards,
Lee
Try here http://www.searchengineworld.com/spiders/
and here
robot agents
I used to have a list of all the Google spider IPs, even categorized depending on what update they were likely spidering for and whether they were the deep, hardcore spiders or just the everyday ones.
But awstats and various other stats programs have started incorporating SE IPs in to their reporting, telling you when and how often you're getting spidered and so that just made me get lazy on it and now that list is lost somewhere and probably out of date hehe.