I just came back later to fully explore my territory.
That's exactly what's bad. Even from a realistic point of view, it doesn't make sense, let alone in the game. Your task was a quick attack - a full-scale invasion, but the mission was only done to make you hurry up and clear your entire territory and gather troops by the required time. Then it's clear why the dwarves managed to gather. Although, again, I don't fully understand Rolf's actions yet.