Gosh, we have all been there. As I was reading your story, I wanted to tell him to leave. Yes, please listen to your inner self as your instincts are always right and you shouldn't have to feel uncomfortable after all these years.