Having followed the immigration debate for the past few decades in various incarnations, I've observed that there's very little "middle ground" between the pro-immigration and anti-immigration viewpoints. Those who are pro-immigration only point out the positives of immigration, and they won't hear or entertain any arguments which might suggest any negatives. The anti-immigration activists are just the opposite in that they'll only point out the negatives.
The arguments also seem to come from different directions and vantage points. In the USA, a lot of the pro-immigration sentiment seems to be based mostly on tradition, that America is a land of immigrants and this historical fact gives us a moral obligation to accept and accommodate immigrants from other lands. This particular argument doesn't really seem to apply to other nations.
Then there are countries like Mexico which expect and demand open immigration to the US, but do not give similar consideration to immigrants to their own countries.
The trouble really seems to stem from the fact that immigration is, in fact, a global issue, yet each country has its own individual policies and attitudes regarding immigration. Some people of a more globalist perspective think that one day, national boundaries will no longer exist, and people will move from country to country just as people move between states in the US today. I don't know if that will ever come to pass, though. A lot of far-reaching changes would have to be made on an international level before that could ever happen.
Until then, there will likely be resistance to immigration wherever it occurs. There will probably also be stronger pressures to assimilate. Many may be accepting of immigrants, but only on the condition that they learn the language and adopt the culture of the country they're immigrating to. It's in this area which I think is troubling to some people. They're okay with immigration and cultural diversity to some degree, but only in measured amounts. If it seems as if one culture is totally displacing another, then this is when it starts to get tense. These are the things we need to keep a better eye on.