Rather, it will speed things up. If you're going to rush, rush hard. More troops mean a faster Game Over. And a non-native 7th dwelling is practically irrelevant; it has no hiding spots, and it's usually impossible to place it next to a Castle within the turn limit. The example with the Bigs is an isolated case. Now, these dwellings are only useful because they provide a path. And maybe to buy a book for your dad. Sometimes you can gather a few single units/resource gatherers. In principle, you can ignore them altogether. They only add variety in very rare cases.