

This logic implies that you would be comfortable with an almost-fully transitioned trans male using the female toilets. They are 6’1", well built, bearded and indistinguishable from a cis male, but they still have a vagina and therefore since we are allocating toilets on genitalia, they must use the female toilets.
No it doesn’t make you a Nazi to say what you just said, but I do think you are worrying about things that really are not an issue.
Stopping trans females from using female toilets does not prevent straight cis men from sneaking inside to do something nefarious – you know, the demographic that is by far the most likely to sexaully assault someone?
I would also say that the idea of changing the definition of racism to mean that you cannot be racist toward a white person is also inherently racist. You could say, “person X is oppressed by people Y, so I understand why they are racist” but that doesn’t mean that they aren’t racist, or that it’s okay.