This is very true. Another way to look at is that it is not something to fear, but simply not indulge it, which is what feeds it. When the dark voices come up, "you're worthless... what a fool you are.... you'll not survive if you lose those things that make you happy," and so forth, to engage in response by worry, fear, shame, guilt, and anger, you simply feed it. And the more you try to shove it away, to disown it, the more powerful you make it. Repressing your dark self, creates a beast a thousand times more powerful, and you are giving it that power from your reserves, depleting your joy in trying to deny it.
The key is to not fear it, but rather simply witness it, begin to recognize it from yourself, and simply let it go, as you say. You own it as part of yourself, then gradually you take the energy that arises from it, and transmute it into the service of good. Releasing fear, vanishes the demon.
I would say if you want to call it a test, that what it is is a gift on your path to expose that you have more work to do.
I know these come sometimes in the form of people, who so severely rub you the wrong way you want to rage at them. That little gift of that person shows you that you still have a demon you need to befriend as quickly as possible! What these "demons" or "annoying people" are, are a projection of our own shadow-self, the disowned, shunned, and rejected parts of ourselves do not want to acknowledge. There are unintegrated, fractured off parts of the self that need healing and integration in the whole.
I don't know how it is possible anyone can truly love another person, if we can't love ourselves. And if we can't love ourselves, then how can we be one with Reality, which loves all unconditionally. Escaping ourselves is not Freedom. Being our true Self is.