Though I no longer technically identify myself with labels like this, I do qualify as atheist in the context of not believing in mythic-literal deities and embracing the modern and rational in the context of science and reason. I don't accept mythologies as scientific realities. I take what can be called an "integral aperspectival" approach which means, I can speak from that position,
as an atheist, while not solely self-identifying as one (or anything else either for that matter).
While I did solely self-identify as atheist for many years and can speak from that context to what you are saying here. Like you as a self-identified atheist I struggled with those feelings of needing to be true to rationality, yet all the while experiencing the ineffable and feeling hypocritical about it. I remember the conflict of identifying myself as atheist while having what I historically associated as "religious" feelings.
I think the important part was for me was to realize the 'religious' experience is not something owned by religion! 
As someone pointed out in this thread citing the "spiritual but not religious" category, that is a good starting place. I later called myself a "spiritual atheist", but ultimately I just dropped labels like this as it too had the connotations
of what you 'should' of 'should not' believe or think for me. Hence why I felt hypocritical. Those of my peer group, those who were self-identified as atheists called such things derogatory
terms such as "woo", when they were hardly that at all.
Now this experience may not be others experience in using the label and that's perfectly fine as long as they are not conflicted. It sounds like you are struggling with what I did, so that's why I share my own experience. There is no reason I see that one can reject the dogma and literalness of mythic systems and embrace the rational, while embracing the ineffable
at the same time. It certainly was true of Einstein and a long list of the great modern physicists. If anything I self-identify as it is a mystic (like them), as the question of God or no-God is really the wrong question at that point. Atheism is a valid perspective, but like all perspectives, its relative, not absolute.
