A couple of quick thoughts on this;
1) put me in the 'I don't love democracy but it's better than the alternatives' camp
I agree, for the most part, but by the same token, I don't see democracy as some kind of sacred cow. I've heard some call it the new "opium of the masses."
Democracy requires a vigilant, aware, and informed populace to make it work. If it's not working, then we have to ask ourselves why. It's a purposeful, conscious process, not something that we can just let run on automatic and hope for the best - which is what most people seem to be doing these days. It's just like people used to say back in the 60s, "All you need is love," but now, it's like they're saying "All you need is democracy."
It's been a constant refrain in foreign policy, too. I remember people saying as far back as the 1990s (about China) "All they need is more democracy, and they would be just fine." We've wanted to push for democracy in the Middle East as well. To some people, democracy is everything - the be all and end all of human existence.
But the thing is, "democracy" is just a word. When we actually get into the weeds of the system itself, looking through the nuts and bolts, that's where the problems arise. Democracy requires faith, and when the people start to lose faith and trust in the system, whose fault is that?
2) there are plenty of functioning democracies who don't have these issues around the transfer of power. I live in one, and have worked in two others (Australia, New Zealand, Sweden)
It's worked in the U.S. as well, for the most part, from 1788 to the present. But I can't believe that no country has ever had any issues with the process from time to time.
I think the main issue with our system in the U.S. is because people worship democracy as if it's a religion, and therefore, they see it as unchanging and with only one orthodox viewpoint. There's no room for criticism or any calls for reform when the political culture is hobbled in such a way. As an example, I would point out the numerous criticisms of the Electoral College and calls for its removal. Yet, we still have the Electoral College, because..."that's just how it is."
3) I think the move from the system you have to a better system is evolutionary, not revolutionary, hence the need for thought and discussion
I agree, but if the rhetoric on social media is anything to go by (and we can even see shades of it here on RF), then how much thought and productive discussion can we actually expect?
I would also mention that, America has been down this road before. I'm old enough to remember when people were saying that everything was getting better, especially after Nixon resigned and the U.S. pulled its forces out of Vietnam. The Civil Rights movement made huge advances, and it looked like we were entering a new age - which was occurring through evolution, not revolution (even though a few people had been calling for that, but most didn't want it). It looked like things were going in a good direction, but somehow, we went off track.
There has been some thought and discussion about that, where people ask "where did we go wrong?" But there doesn't seem to be that much interest in the deeper questions about the state of affairs. Most of the public's attention seems more geared towards soap operas and melodrama in politics, where anything with any real depth or thought is just too boring for a lot of people.
4) I think part of the answer is in sensibly expanding political power away from a two party system, and I don't think the current US system practically does this
There's nothing legally to prevent people from voting for whichever candidate they wish. It's not really a systemic issue as much as it's an issue within the political culture itself. There's a palpable battle being waged, and it's more of a cultural battle than anything else - within the political culture. In the end, people tend to hold their nose and settle for the lesser of two evils and try to get on with their lives as best they can. But over the long haul, it slowly wears and whittles down people's standards and expectations regarding the political system. As a result, they lose faith in that system.