The supposed advantages are nothing comparable to an FTA. Just as importantly, all the Commonwealth countries are thousands of miles away, whereas the EU Single Market - of 400 million people - is right on the doorstep. In commerce, freight cost and supply lead time are big factors in who you do business with.
And there is nothing morally despicable about permitting cheap labour to come, to do the jobs the indigenous people do not seem to want to do. The Poles and Romanians come to earn more than they could at home and they are full citizens, enjoying the protection of UK law and labour practice, which are also better than at home. The NHS and UK fruit and veg farmers are already finding out the hard way what happens when you make these people feel unwelcome. The Brits won't do it or don't have the training.