I do agree that brain activity is all according to physics, but I don't see that as being contradictory to the presence of a unified sense of self that makes decisions.
Psychological experiments such as those done by Daniel Wegner -- referred to in his book
The Illusion of Conscious Will, and other such data reveal that human beings are often mistaken about their own intentions, invent false intentions in retrospect, or become mistaken about whether an event was caused by their own volition or an external agent. Studies done by Libet further reveal that we are mistaken as to when we make a choice -- brain activity precedes the sensation of choice and can predict the choice to be made with a significant degree of accuracy.
Besides this, there is still no center of consciousness or self center within brain activity. It is a machine doing many different things throughout in parallel with no unifying center. That is why I reject the notion that the self is doing anything. The self is an illusion, and our perceptions of our intentions are often inaccurate.
It is not a unified self that makes decisions, but an entire organism with an active brain and nervous system from which volitional behavior proceeds. It can't be tied down to a specific self or agency unless the self is identified with the whole of the organism, which still doesn't help much when looking at what actually happens in the brain.
I do not deny that consciousness exists. Perhaps it is a by-product of other functions. But I do not believe it to be a causal agency.
The Illusion of Conscious Will, by Daniel Wegner does a good job of arguing that the conscious perception of choosing happens when somehow brain activity has accurately predicted the next action. When this fails to happen, a sense of free will decreases.
Because the activity of the brain is so intricate and not tied down to any specific area of the brain, I do not see how a self could be choosing any actions or why the brain would need one to. Volitional behavior proceeds from too many sources within the brain and body to be tied down to one specific self.