I get your point--and agree--yet I have heard many atheists say how free they feel after letting go of notions of being accountable to a deity and its rules.
Absolutely.
There's no one simple answer for that, either, but I would agree that it's a common enough state of mind. I would see a few reasons for it (in basic terms);
1) Atheism, or at least self-described atheism, can be simply an act of rebellion of theism, due to almost anything from the trivial (but I don't WANNA go to church each week) to the profound. (eg. My church decries homosexuality, and I am a closet homosexual, therefore I will reject the church, and religion).
2) Atheism does represent freedom from both a deity, and (usually) religion. So there is no external party, be they divine or human who is telling you your every action will be judged, that eternal reward or retribution are tied to them, that there is a global struggle, battle for souls, or that everything makes sense, etc. It is, in some senses, a simplification of life.
3) There is, I am sure you'd agree, a period of cognitive dissonance a person must go to before moving from theism to atheism. There is not, for example, the possibility of a personal revelation changing someone's mind immediately. It's tension between what the person believes is true, or perhaps what they believe is right, and their current state of belief. At some point that tension becomes untenable, and a decision in one direction or the other is forced. It is completely normal to feel a sense of freedom once a decision is made. I would think a theist who has a crisis of faith, but then has their faith truly reconfirmed (ie. not lip service, but honest faith) would experience similar relief, but would couch it in different terms.
4) A 'new' atheist would first realise what they don't have, mostly. They don't have the restrictions and rules of religion. They don't have the same community of people (which may of may not be bad), they don't have the same expectations. This is not a meaningful or healthy way in which to live life, imho. Atheism is NOT a structure or framework for life in the way religion is. Instead, it's the denial of that structure (I'm talking very generically here...I know). You'd hope that the structure is replaced with something, be it an atheistic religious belief, some sort of humanist view, or pretty much anything else that indicates a level of thought and self-awareness. The chance to move from a religious framework which has possibly been tightly defined and chosen for them to a position where they can explore and determine a path, or make up their own is certainly freeing.
=============================================
I think it's more the implication that 'freedom' equals 'lack of responsibility' that kinda sticks in my craw. I have the freedom to make whatever choice I want, only if that freedom is compared to some sort of objective morality imposed by religion. But, from my viewpoint, morality is subjective, regardless of whether an individual thinks this is true or not. Much like a theist would suggest God exists whether I believe or not. Theists are just as 'free', but have identified, or been coerced, or feel connected to a religion that provides structure, and morals, etc.
The assumption inherent in the quote Thana provided earlier is that the removal of that structure equates to some sort of removal of morals, which I see as a fallacy of the highest order.
Can atheists be immoral? Yup, although they probably have the ability to twist moral language in a more flexible manner to 'justify' their actions. Can theists be immoral? Yup, although they probably have the ability to throw themselves on the mercy of their religion, and claim to have reconnected/rediscovered their faith.
Anyways...I actually think this stuff is really interesting and valuable to talk about. I just think such a discussion needs to be grounded, and avoid trite point-scoring.
And no, I'm not directing this at Thana. Hopefully she realises I'm not flogging that dead horse. It was just one quote, let's not get crazy. But over-generalisation is a hobby-horse of mine, and I TRY to apply the same standards, be it atheism or theism. I've called atheists on it before, and would do so again. I've called Deists on it too (although I suspect Sterling Archer might not agree with my point of view...lol).