If the guy isn't scared to lose you and isn't fighting for you. It means he doesn't care about the relationship with you. So, he is not the one for you, and walking away from the relationship is the only right choice.