Prohibiting free speech--including hate, prejudice, and bigotry--does push people out and does create these "gatekeepers." Now, bullying is one thing, which should be discouraged, but a Klan rally, which is not only Constitutionally protected, allows them a more safe and structured environment to speak their mind, peacefully blow off some steam, and return to life. Should they not be allowed to speak is when their grievances begin to have some truth and merit to them, especially when they rightfully claim their rights are being denied. And it's not a good position to be in when you have banned nothing more than words.
When the Klan can walk your street, the idea that black people have and should have the same rights and liberties as white people becomes debatable. It becomes a fight and fights are largely won by three things: money, number and power. Minorities, especially historically oppressed minorities, have little of those three things. Another thing to keep in mind is that no fight is without it's pains. Though black people might win the fight against anti-black racists, at what cost will it be for them? How much of their political, cultural and social freedom will they have lost to the attacks of those racist folk and organisations which the rest of society protected even though they disaproved of them. It's very easy to welcome a fight when you aren't on the first line and unlikely to suffer from it.
What is banned isn't "mear words" they are ideas. Ideas that not all races and ethnicity are or should be equal; the idea that one faith should dominate over others; the idea that a gender is superior; the idea that some sexual attraction and identities are sicknesses. These ideas make a mockery of "free speech" by their very nature. These ideas are poison and completely incompatible with liberal democracy. What hate speech law tries to prevent is allowing racist people (for example) the breath and liberty to not only express their beliefs by harrassing people, but also the opportunity to organise. The KKK is a very dangerous organisation with an anti-democratic agenda. The racist people who compose it are individually not a threat at all to society, but together with the room aforded to them, they can organise, find themselves a leadership, pull resources in common, concentrate their energy unto specific issues to advance a wider cause, that of racism and racialism.
PS: the idea of physical catharsis for anger like sreaming insult, destorying stuff, menacing, etc. can help reduce and control anger is bull****. It actually increases it. If you let racist people scream racial epiteth and march in the steet under some banner and rallying symbole, you only make them eager for the time they can do it again and harder because they get an immense "kick" out of hurting those they hate. The human brain loves a fight and the sensation that what it hates and fear is suffering. They basically nurture their violence and hatred in addition to making their target fear for their safety and their rights. If I was black and living in a city where the Klan hold rallies, I would make myself as small as possible; how "free" am I in such circumstances?