Using the verb to accuse sounds like an excusatio non petita.No you didn't. You've accused him of being Muslim.
If you tell me " I think you are a Buddhist" or "I think you are an atheist", it's not an accusation.
Being a Buddhist is something absolutely normal and nice.
So you're not accusing me of anything.