Stop feeling guilty when I stick up for myself

Don’t get me wrong, I don’t take anyone’s sht and I tell it like it is. But most of time when it comes to my really close friends or family I try to bite my tongue. And I hardly ever stick up for myself. There’s one friend who is constantly saying this one thing to me that really gets at my nerves, about my age and maturity. This friend thinks nothing of it. But it really irrates me and makes me feel bad. And she doesn’t even know it because why? I won’t stick up for myself because I don’t want to hurt her. Even when my friends dck me around half the time I blow it off and act like its nothing because I feel like ‘oh, I might hurt this friend, and then this friend will be unhappy’