It's not that important: in any case, rebuilding a town is cheaper than building it from scratch.
Or is the multiplayer cheat, where one of the opponents has a town of the desired faction nearby, while the other doesn't, also better?
Overall, I think there are more advantages.