Castle, Tower. Optimal combination of Price/Quality/Magic/Combat.

Regarding the Tower, I have to disagree, as it is the most resource-hungry town, and fully building it up requires a great many game days. Only the Giants are very cheap there, but their cheapness is comparable to their crappy quality. Upgraded Mages and Titans require a huge amount of resources, and if playing on the maximum difficulty level on a small map, the Tower will lose miserably even to weak Barbarians and Undead, who also fare poorly on small maps. And the fact that the hero literally has access to fast (Gargoyles) and shooting (Master Gremlins) units from the start provides no special advantage over an opponent, only over neutrals... and even then, if you play as Solmir :)
The Castle, however, is of course ideal for establishing yourself and is maximally simple to develop. There, you don't build the Blacksmith just for beauty, etc.... in general - "cheaty," as they like to say here :)