Hey, roads, dust, and fog.

On fast maps, a convenient road junction plays an even bigger role, as well as the number of objects scattered along the roads. It also becomes much harder to play if you get stuck on a one-way road, as while you're wasting time going around on the off-road, your opponent can comfortably spread out and take over everything on the fast roads. And redesigning one-way roads isn't a plus on all map templates, and not all players support it, and according to the rules, standard one-way roads are still the default.
Another significant drawback of fast maps is that if someone gains control earlier, they can easily take over all the other castles in the same turn just with ranged units – this isn't right.