I believe he did visit Ishmael, since this features in the Midrashim. I don't see any Midrashim about him visiting his sons from Keturah. But that doesn't say much, since he may have sent them off towards the end of his life. But that's neither here nor there.
Right. Since G-d told Abraham to listen to Sarah's command to send Ishmael out, the purpose being to separate him from Isaac, the only way for Abraham to visit Ishmael is to do so alone.
I'm sure at some point he did. But at least, by the time he sends out the sons of Keturah, Ishmael is not living in Abraham's home. And for the important part: when the verse speaks about the fulfillment of the prophecy of him living among his brothers, it mentions it in connection to his living in the Arabian peninsula and its environs. Who cares if Ishmael had or had not spent some small or large amount of time near Abraham? The point is whether "among his brothers" referred to Isaac, not Abraham. I don't really get why you keep trying to push him living near Abraham as though its a game changer. We've both brought a Midrash that mentions that he did. I don't understand what that has to do with anything though.
It seems like you're trying to ignore what the verse explicitly says (Ishmael living in the Arabian peninsula in connection with language that mirrors the prophecy of him living among his brothers), in favor of a contradictory implicit statement (that Ishmael lived near Abraham because they loved each other) that you'd like to pull out of it.