Haha, for this, you just recruit a hero from a Stronghold, and with him, a unit that removes penalties on stones... Why join someone? What's the point of such units anyway? You still get penalties regardless.
Yes, but you can reach them freely, and you might save time if you compare using an expert pathfinding skill and wasting two days (approximately) on movement.