I'm not sure this is about skin-colour as much as it is about cultural values.
If you are part of a very conservative culture you don't want your children affected by those with more liberal values. A culture can only continue through its children, so if those children are disappearing into a different culture (value system) then the parent culture is being eroded. So you keep them apart.
Yes this is a good point and I found myself thinking about it too while answering the way I have - especially in the context of Judaism which
@Harel13 brought up.
But I would say two things:
First, in my view you have no business using your children as a means to pass on a culture that you value. By all means teach them your culture in the hope they may value it (we all do that), but you have no right to prohibit them from contact with other cultures. When they are adults, they must choose what is right for them.
Second, you have no right to use skin colour as a lazy proxy for culture. All of us in the "western world", I imagine, know plenty of people of different skin tones who we find to be congenial companions, with cultures either the same as ours or only different in ways that are not problematic.
In the S Africa of old, there was a problem of culture, because apartheid was designed to ensure no black ever got educated or was able to get familiar with the culture of the white colonists. To put it crudely, they were kept in a state of civilisation a century or so behind the whites they served. That is changing now and a black middle class exists, just as it does in say the UK, France or the USA. So the cultural gap is being closed.
Cultural differences are real and not to be underestimated. Nobody says you have to socialise with people from a culture that does not fit with yours. (In my own family, one sister-in-law comes from a family originally from India in which the father disowned her when she married my brother, because he thought he should have the right to choose her a husband. She was a British, university-educated doctor and told him where he could shove that idea.) So, sure, be mindful of culture differences and step round them sensitively. But do not refuse to have contact with people on the basis of skin colour, before you know anything about them. That is prejudice: racial prejudice.