Hard work is no guarantee of any large income. No work at all can still lead to affluence given the right circumstances.
That's not to say that hard work plays no part whatsoever in high incomes. It can. It's just no guarantee.
The fact that Luck has more to do with high income than hard work makes a lot of rich people uncomfortable, especially the ones who attained affluence by climbing out of poverty through hard work. People in general don't like having their senses of agency questioned or denied. But the environment a person was raised in, the type of mind a person was born with, the events that appear before certain people, the ability to spot, and take advantage of, opportunity... all of these are Luck-based, and mean everything.
People also don't like it when they work really hard to attain something, get it, and then see someone else get the same thing by doing comparatively little. It's "not fair". We also get frustrated when we see other people struggle with something we find easy; "I can do this thing easily! Therefore, there's no reason you can't!" This, of course, is basically assuming that everyone is basically the same in terms of ability, which is unwilling to accept the fact that certain people struggle with things that other people find easy. As a person with asperger's syndrome, social matters don't come naturally to me as they do others; everything I say and do in social contexts needs practice, and has to pass through some other process in my brain before it can be expressed accurately. I've compared that to multi-lingual people who speak languages that aren't their native ones.
So, even if it comes from a place of empathy, the idea that hard work is all that's needed to be rich, and accusations that poor people are lazy, is elitist, toxic, and in certain cases, dangerous.