They didn't have to "leave" in order to change languate and culture.
Indo-European language and culture didn't make anybody "leave", just like Christianity didn't; we're still the same people. Remember that "Celtic" is a linguistic term, so if Austrians don't speak a Celtic language, they're not Celtic people.
I have Irish ancestry, but my native language is English. Therefore, since English is a Germanic language, I'm not Celtic; I'm Germanic. Just like every other person who speaks a Germanic language.