Because queerness (trans, gender non conforming, gender fluid, agender, bigender and related) threatens hierarchy.
In western society regardless of how ‘progressive’ some parts of it have gotten, for the majority there’s still a strict hierarchy. Man most important, then woman, then children first boys then girls. Trans people completely disrupt this hierarchy by being able to change what they are and those who cling to hierarchy freak the fuck out over it.
Then there’s the sexual panic, a straight man who’s insecure is gonna freak out if the woman they think is cute actually has a penis.
If the transphobes thought trans men were men then your comment would be accurate. But they just see them as confused women and easy to just ignore them like they ignore cis women.
(You are right about trans women bearing the brunt of the hate, and I think so much of that is sexual panic from cishet men about finding a penis owner attractive)