Well, since most arguments have already been fleshed out pretty well by the people who've already posted, I'll only make a slight addendum of sorts to what Rat wrote out (well done, by the way).
I think the aspect of rap that makes the people question it's authenticity as a form of music is probably the fact that rapping in and of itself is more related to the arts of oration/rythmic declamation than it is to music. In fact, a rapper freestyling without a backbeat is probably gonna be closer in experience to that of listening to someone making a lecture, a poetry reading or a sermon(albeit a more lively one than most I've attended

). Add a beat, and voilĂ ! It becomes Hip-hop which, in opposition to just rap, is purely music if you remove the rapper. Of course, certain artists tend to be more borderline between signing and rapping, and in these cases I guess I consider it to lean more towards music than the preceding examples I've given.
So, as the ballet or the opera, one has to be ready to adapt to a hybrid form of art if they are to appreciate rap. I for one don't care much for either of these three forms of art, as I am solely (or almost) interested in music. I'm not saying they don't have merit, just that they aren't really my bag.
Now, as to wether saying rap isn't music is racist or not, I think Dr. Uzuki said it best when he said that we can only tell on a case to case basis. I consider rapping to be a form of artistic oration by itself (so not music) but I do consider the constructions these guys use to rap on music. Does that make me a racist? I sure hope not. Now, If my only argument would have been "them's niggers, they can't make music" or something along those lines, then certainly the answer would have to be yes.
However, I think in most cases people who don't like hip-hop tend simply to find themselves alienated by it in being unable to relate to the experiences these guys are sharing, as Rat said in his post.
Jam it back in, in the dark.