I agree.
If God is defined as Creation, what can be more proof of Creation than a new creature as an infant?
Well, as an atheist, I don't believe any God exists, so that's not really why I don't like the statement.
And, as HerDotness pointed out, even if the baby is a creation of God, that doesn't equate to the baby knowing this.
That doesn't mean that the infant has any clue of what "God" means which is obviously the reference of "We all begin as atheists."
I don't understand why people would be upset at the idea that everyone starts out an atheist. "Atheism" simply means not believing in gods. An infant doesn't have the mental maturity to form any kind of belief, so an infant obviously doesn't believe in God.
While it could be argued to be technically true, I think it's a silly argument just designed to raise the hackles of people. Labels denoting beliefs should be reserved for when people are actually old enough to understand the belief, and decide whether they hold it or not-- whether we are talking about Christianity, conservatism, environmentalism, or atheism, etc. I truly feel it's meaningless to speak of "defaults" in such a matter, and a disservice to the child to slap a label on him before he even knows what he's thinking.