I'm not telling who you're attracted to and who you're not, I'm not telling you you'd get violent or call someone names. I'm not saying you're prejudiced against trans people, gay people, whatever.
If you call someone ugly, and you mean it, that doesn't mean anything more than you're a knob. If you're saying it because they're clearly gay, that'd be homophobic. Trans, transphobic. Because they're of a different ethnicity, racist.
I'm saying that if your sole reason for not wanting to engage with someone, sexually or otherwise, is because they are transgender. That is transphobia. You may not find anything wrong with that, that's fine, whatever, you're you. It's still transphobia. Same as it would be racism if the sole reason was because of the colour of their skin, and everything that means to you.
I'm not having a go, telling you what to think or not... I'm just laying definitions and examples out.