Well, it's not a given. Nature can also build up to level 2 and upgrade magic to level 2, and it won't be easy for the Barbarian...
And Haven at the beginning... well, if diplomacy or development is at least a little developed, then there are already chances, and it won't be so easy with the gold golems, and even hobbits can be very effective if they get accuracy.
I'm talking in general... of course, in skilled hands, even air can burn...
P.S. Are you going to argue that the strength of Barbarian units is inferior to any other?