It takes time to move your life around, and to disassociate from negative people. However, in the long run it is better for you and your self-esteem to be positive, think positive, and to surround yourself with positive people. Negative people create and thrive on drama, they blame others instead of accepting responsibility, and they do things to bring others down – misery loves company, and toxic people/relationships all eventually cause pain.
It can be hard to slim down the people you associate with, but in the long run we are all better off having a few really good, trustworthy, supportive friends than a hundred backstabbing leaches hanging around.
Everyone needs to do what is best for themselves and their own lives.