If you're near a rock that is on the side of a mountain or cliff, first get higher ground using a sword wielding character. Double jump so that you begin to glide and then strike down with your attack (Plunging Attack) onto a rock. It'll break in one hit. Crystals are destroyed in two hits. Otherwise, if you have a character that can wield a Claymore you can destroy it in one hit.