The thing I don't get is... how does the black guy think that by calling you a racist anything is going to change?

If you were a racist, calling you on it is likely to simply reinforce your existing feelings or stereotypes.

If you're not a racist, but aren't attracted to someone of his type, calling you a racist isn't going to make him more attractive to you.

So it seems like a lose/lose proposition no matter what. Stupid.