To swap out your party members, it's not simply a matter of going to the menu. Instead, go into a town and look for a Bar/Tavern. Talk to the Tavern Keeper and the option to Change Party will be available. From there you can swap any current party members with an idle one. Party members do not gain experience when they're idle but they level up fairly quickly once active.