Mammals have only been around for 125 million years. Not billions.
We started with very small reptilian brains. From 125 to 65 million years that brain size and complexity increased by more than double. In addition the exact nature of early mammal metabolism is questionable. "warm bloodedness" is not a clear line between hot and cold.
The Tenrec for example has a highly variable body temperature. Night and day can swing the body temperature more than 10 degrees.
Naturally not all Dinosaurs are quite the same, the exact nature of their metabolism is unknown for many groups. For those groups who are known to have high-metabolisms there is a trend in brain size increase over time.
In mammals post 65 million years brain size fairly explodes in most lineages. Humans exaggerate this trend but not right away. We have slow growing brains for more than half our history. There were two 'growth bursts' likely tied to mutations, one seems to have rid us of our saggital crest (though this one mutation alone is not enough).
To step away from "warm blooded" animals, one can hardly argue that the modern cuttlefish and octopus has a brain remotely like the brain of basal mollusks like clams.
Big brains are very expensive in terms of metabolism. If you only need to eat once a month you can only 'afford' too much of a brain. And you only have to be smarter or more patient than your prey. Considering how much the 'cold blooded'/'small brained' animals still dominate this planet, truly big brains are frankly not all that needed.
We use our brains mostly to have highly structured complex social lives and to imagine new tools. What does a croc need that for?
One last point, that I made early on. It isn't simply about brain size. It's about brain complexity. Please look at my point on the Tenric and the Marmoset. The Tenric has a brain more than twice the size of the first mammals, but its organization is roughly the same. The Marmoset uses the same brain volume for very different tasks. This is what causes the vast difference in "intelligence" between the two.
Much of human evolution isn't all that adaptive, but other forces like Biased Gene Conversion.
PLoS Biology - Hotspots of Biased Nucleotide Substitutions in Human Genes
wa:do