Even if it’s 1-5, but if you can upgrade without rebuilding in the city, it will be different for everyone. For example, an upgrade for a Tower will be so-so, but for a Castle or Rampart, not to mention Elementals, upgrading them early on gives you a significant bonus. Although, for some, it might be quite useful, like the Swamp, which has an unjustified upgrade for Gorgons through a Warehouse; I don’t even know. Personally, I’ll stick to my opinion that it’s better to leave the Fortress alone and not touch it.