It is always better to keep your relationship with your affinity rather than getting a new one. In order to maintain your relationship with them, you will have social interactions throughout the game, and there will be moments that your character will encounter which will be decisive whether it will gain your relationship to them or the opposite.