To be honest, I thought the penalty applied to the hero as well. Well, in that case, it's not bad for collectors to give away one gnoll.
But sometimes you still have to move around with an army that isn't from the Fortress. Diplomacy is still a useful thing; it's better to take a movement penalty but have a good army.
Auto-translated
