Yes, I'm worried.
For one, I don't believe all the talk about this being no worse than the flu. The behavior of the countries concerned just isn't consistent with that. You don't lock down 50 million people just for a new strain of flu. You don't see the disruptions of world travel, the Chinese cities with empty streets, the quarantine camps, the military roadlocks...
So I'm inclined to suspect that the Chinese government knows that this is a lot worse than they are saying. Their biggest priority is preserving social stability and the survival of their regime. So they will do anything possible to prevent panic and civil disorder.
Thus, I don't really trust the official numbers. Rumors out of China are that number of cases is perhaps 10x what we are being told, that people are dying in the streets, and that crematories are running non-stop.
I suspect that the fatality rate is much higher than the official 2% or whatever. We don't really know how long the disease typically lasts once somebody gets it. Seemingly more than a month. So presumably most of those infected in the early days of the epidemic are still ill. We don't really know what their long-term prognosis is.
If we divide the (reported) number of deaths by the (reported) number of cases, we get the widely reported 2% number. But if we divide (reported) number of deaths by the number of cases whose final outcome we know (deaths + recoveries) we get fatality rates ranging upwards of 10% to as much as 90%. (Depending on the numbers available, which I don't believe are reliable, though my guess would be towards the low end of that range. That would be consistent with SARS, which was harder to catch.) Of course if many of the survivors only survive with the help of hospital intensive care (according to the CDC, up to 1/3 of all hospitalized patients will need intensive care ventilation), rates will go up as hospitals are swamped and more people just die at home.
So I don't know what to think, but my gut tells me that it's a lot worse than the media are telling us.
Of course a persuasive counter-argument is that the behavior of most world leaders (outside China) isn't consistent with their believing in the doomsday scenarios. I think that most of them still seemingly trust the WHO numbers which essentially are the official Chinese numbers.
So the tip-off will be when some countries start to go into crisis mode similar to wartime, and when the leadership class around the world starts disappearing into secure guarded facilities. Then economic collapse, empty stores shelves, no more public services, martial law...
For a very realistic portrayal of what I worry might happen, watch the movie
Contagion. We are already seeing it starting in China.