As far as I know, the Roman legions were the first people of the Mediterranean region to switch from tunics to trousers, I believe around the 2nd or 3rd century AD. However, one could argue that pants had been known among the ancient horse nomads of Scythia and Sarmatia for a few centuries before that.
In general, people who ride horses tend to favor wearing pants, for reasons that are probably intuitive to anyone who has ever ridden a horse in a skirt or dress.
Of course, the manliest clothes of all are puffy shirts, tights, and high heels.