Hero engineers.

I rarely use siege weapons. If a hero is a good Chaos Knight, he doesn't need a catapult at all. Only a fool would hide from a Chaos Knight in a castle. And even against a specialist in dark magic, the opponent won't feel comfortable sitting in a castle. And towers and walls are destroyed by the Crystal of the Secret, which, unlike the catapult, can destroy the central tower immediately.

Siege weapons are only good at the beginning. But many factions have heroes who can easily defeat neutrals without them. In the late game, any siege weapon, including a catapult, is destroyed with a single shot.