When they start attacking your server, then you should really get worried, or when you see them spoofing your content, which can be done without even entering your site... you'll end up with huge bill for outside content providers, that you have no idea where they came from...

Personally, I think spoofing content is the most worrying thing...