Staying clear of them or having some space is not being selfish, it’s called self-preservation. I too had friends like that along the way and I thought I was being a bad friend if I wasn’t there for them. But I had to pull away or else I was being dragged down under with them. I was of no help to myself - the boat sinking without the life raft- situation.
It’s good to create boundaries.