Every year since I was little I'd visit my gran in a working class suburb of Rotterdam.
It was government subsidised housing, so was low income and probably about 95% were white. It was fairly safe though, with a good sense of community. Someone in the block of flats would always look out for my gran who lived alone, and us kids would happily play outside without supervision. As with any poor area there was some anti-social behaviour, but not a great deal.
Over 20 years, due to cheap housing, the area became probably 80% immigrant and very unsafe. Gang violence, drugs, vandalism, robbery became common, it became very noisy and dirty. My gran would not go outside after dark, and people tried to break into her house multiple times. No one would look out for her anymore, many of the people in her block couldn't even speak Dutch.
When she was younger, after her sister died in childbirth, my gran had helped raise her mixed race children at a time when attitudes towards such things were a lot more prejudiced than today, and she nearly remarried to an Indonesian man. She was by no means a xenophobe, she would just take people on their merits.
However, based on her direct experience, she saw a connection between immigration and increased crime, and, while she wouldn't have used terms like "replacement" she did feel a bit like it was turning into a foreign country. She certainly saw immigration on that scale to be harmful and the net effects to be negative.
One of the reasons why such rhetoric resonates with many people is because it matches their own experience. When politicians and others who live in wealthy, low-crime, mostly white suburbs tell them that it is racist and xenophobic to not be entirely enamoured with their experiences of mass immigration, it somewhat stokes their resentment, hence we see a backlash in many Western countries.
(and fwiw I'm an immigrant married to an immigrant
)