I finally found it!
It's Philo, Special Laws 4.178. I even found a few comparable texts:
The ancient references for religious ostracization are (1) early Christian persecution in Pliny, Ep. 10.96; Tacitus, Ann. 15.44, and Suetonius, Nero 16 (2) Jewish proselytes in Philo, S. Leg. 4.178; Tacitus, Hist. 5.5.2 (3) Christian alienation from families in Justin Martyr, Trypho 2.2-7.
The infamous text that I had to search for is:
"(178) In the case of the first, because he has made his own kinsmen, whom alone it was natural for him to have as allies and champions, his irreconcileable enemies, by quitting their camp and taking up his abode with the truth, and with the honour of the one Being who is entitled to honour, abandoning all the fabulous inventions and polytheistic notions which his fathers, and grandfathers, and ancestors, and all his kindred, who cleave to the beautiful settlement which he has forsaken, were wont to honour. In the case of the second, because he is deprived of his father and mother, his natural defenders and protectors, and by consequence of the only power which was bound to show itself as his ally. And lastly, in the case of the woman who is a widow because she has been deprived of her husband, who succeeded her parents as her guardian and protector; for a husband is to his wife in point of relationship what her parents are to a virgin."
