In the context of the normal position being Israel it precisely does mean moving them back to Israel or removing them from those other lands.These words have different meanings. Remove means take (something) away or off from the position occupied whereas recover means to bring back to normal position or condition.
Actually remnant means the remainder or as you said, what is left. Since it did not say the part of what's left will be recovered but rather that what's left will be recovered, you are pulling the wool over your own eyes to suggest that since it does not use the word entirety it is not referring to the entirety.Even if it was interpreted to mean the remnant of Jews in these places would be recovered and then move back to Israel that does not mean that the entire remnant of Jews (every single Jew) living in those places would move back to Israel.
If I were to say, I removed what was left of the cookies in the jar it would be understood that I was reffering to all of what's left, not a part of it. There is no ambiguity here, you are clutching at straws that simply do not exist.