How does it go against fidelity? Love and sex are not the same thing. They certainly can (and of course often do) go hand in hand, but they don't have to.
To think that sex must only be a practice with your spouse is absolutist and I don't see any argument for it.
Infidelity is wrong, because you're hiding something, you're being dishonest to your spouse, but nothing is hidden in swinging. I don't see how that's unfaithful.