I hear you on the guilt by association. But you don’t ever need to apologise for being born who you are. What you do with your life is what matters the most. As @offendicula said, you can’t control other people, you can only control yourself. Being a good role model to and good influence on other men and boys is one of the single best things you can do. :)