Changing the members of your party is not straightforward and there’s no UI or button that allows you to do it. In order to do so, you must first set up camp. Talk to the party member who you’d like to remove from your party. Choose the dialogue option that says “wait in the camp” or something similar. Next, talk to the party member you want to add to your party and ask them to join you.