What can i do if my competitor do spamming activities for my project?

Hello all i have a question for all seo guys, As all know about Google Penguin Updation If my site rank high with a competitive keywords but in few days its going some drop, when i checked the reason i found some one i don't know who is this doing spamming work for me and takes many links in unrelevant and adult site for my website. Please let me know what i can do my website is going down day by day and how can i take action for this and inform to Google its not my work. I think this is a very big issue for all SEO guys .