So I should compile an "Army of the Short".
I don't think there should be physical restrictions for joining the military. However, if one is unable to perform their duties due to a physical deficit, they should be removed from the military, or at least demoted depending on the nature of their deficiency. One outstanding instance where the military should discriminate in admissions based on physical criteria, could be the prohibition of those under 5'4 from becoming air force pilots as the safety restraints in the pilot's seat are too large. That does make sense.
Many of the reasons are just as simple as the one you listed, but it also goes into psychological profiling as well. I think from a team building perspective it is a complete and total distraction (not that there is anything wrong in my view of transgenders). It takes the focus away from gearing up for war to social constructs of who is called what and what is acceptable. By the time you're done figuring out if you are Sir, Ma'am, or Ze you're dead. That's why this crap doesn't work on the battlefield...