If, on a large or medium map, you delay building your Castle until the very end, it will likely result in a loss due to a shortage of resources. If you're on a small map and aiming for Archmages, that's even better.
I agree, it also depends on the size and duration of the game. But I usually play on large and very large maps.