After all, you'll probably gain more from it!
I don't think it's worth building a Fortress (as I understand it, it costs a fortune just for a bunch of ghosts) and ghosts in the first week; it's better to build Vampires and a Bone Thrower.
Fortress, ghosts, Stronghold, and Castle, in my opinion, are better built after the Capitol.
At the very least, Fortress, ghosts, and Stronghold should be built in the second week.