Mark at IntenseCash has run into this a couple of times. You should check with him, I think he said he's had good results by simply contacting the offending affiliate and in so many words saying "this isn't cool."

I definitely think it's shady and whether or not it's in your TOS, I still think it's wrong for an affiliate to do that.