You can build a town hall earlier, around the middle of the second week! And for the Barbarians, by the end of the first month, they will have all dwellings except for the Cyclops' and almost all creatures upgraded!
Even with extremely bad luck, you won't have Ogres by turn 200!