Having to wear a mask should really be the least of anyone's concern. Is it really that big a problem for most people? There are obviously legitimate reasons not to wear one, but I don't think anyone is criticising people in that category and I'd never judge someone in a shop (for Eg) for not wearing one because I wouldn't know their personal circumstances.
The main reason for wearing one is to protect others, it's really that simple.
Those whose attitude is "well I don't care, I'm not wearing one because you can't make me" sound like spoilt children. For the majority of people it's the tiniest inconvenience in order to reduce the risk of someone else, potentially someone more vulnerable, from catching a nasty disease.
If you are easily able to wear one and just can't be bothered or are trying to make some sort of pathetic "protest" about having been in lockdown for so long, then being labelled as "selfish" isn't unreasonable I don't think.