Its parents choice to give birth to.. baby, NOT certain baby.
So, why responsible ?
Not sure what you mean by "NOT certain baby." The parents choose to give birth to a baby. One thing you mentioned was a comparison of those born in rich families versus poor families. If the parents already know that they're poor and choose to give birth to a baby, then those conditions are solely on them and not the baby. It's the same if they're born in a peaceful country or a country ravaged by war and terror. If they're old enough to have sex and give birth, then they're old enough to know where they are and what the conditions around them are like.
In the case of those who are born physically challenged or with some sort of birth defect, there might be some genetic reason which wouldn't necessarily be the parents' fault (although that would depend on various factors).
If you're looking for some sort of spiritual or religious explanation, there's an old trope about parents blaming it on some "family curse," although that wouldn't necessarily be considered due to any personal fault or blame on the baby itself. Or they might think it's some punishment from God over some sin in their past, but the parent might think it's their own sin, not any sin attributed to the baby, even though the baby still has to suffer for that sin.
Parents might blame each other, such as we see here with Loretta Lynn and Conway Twitty blaming each other for their kids being ugly.