I'm sorry that your DH was called out by people for wearing a mask. It's upsetting that adults struggle with accepting a choice as simple as mask wearing and keeping their thoughts to themselves. It feeds over into their kids, who will then pick on mask-wearing classmates at school. It's not like someone else's mask is affecting them. Definitely not silly to work in solidarity with your family but it is very silly to disparage others for masking.
I can see both sides (you and your DH, not the mask shamers) and I suspect you and DH will have to discuss and maybe even agree to disagree. At this point, DH and I, who have also agreed on most everything throughout the pandemic, are following the science and making mask decisions individually based on our situations. I probably mask more in public settings than he does now since I'm with DD(9) more. She prefers that I wear one - it makes her feel better and doesn't hurt me. What do your kids think? If they have an opinion, it would be worth a polite convo with DH about showing solidarity and hearing them. If that's the case, perhaps your DH could have a few succinct responses ready to shut down the disparaging comments.
Mom to Two Wild and Crazy Boys and One Sweet Baby Girl