The number of recruitment camps and mines varies in the controlled territories, and this is where the balance lies. The different strength of the garrisons and the different size of the territories mean that different races emerge into the world in different ways. Orcs are almost immediately an attacking race, while the others are more defensive.

Are Elves a defensive race? They have incredibly powerful archers of level 3 + druids under the protection of thick ents. + 2 melee units with mass attack, one of which also has no retaliation.
In my opinion, they don't lag far behind (if they lag behind at all) the Orcs in the speed of clearing the map.

Knights are also quite an attacking race. Not as fast as the first two, but not far behind either.

Dwarves, perhaps, but considering the ballistics for absolute defense, they also perform quite well in clearing territory. But in general, yes, they are slower than the other three races.