Hero engineers.

I rarely upgrade siege weapons. If a hero is a good Chaos Knight, they don't need a catapult at all. Only a fool would hide from a Chaos Knight in a castle. And against a Dark Magic specialist, the opponent won't feel comfortable staying in a castle. Towers and walls are also destroyed by the Crystal of Secret Knowledge. Unlike a catapult, it can destroy the central tower in one shot.

Siege weapons are only good at the beginning. But many factions have heroes who can easily defeat neutrals without them. In the late game, any siege weapon, including a catapult, can be destroyed with a single shot.