Ballista and towers.

The amount of damage depends only on the number of "structures" (buildings) currently in the city. The level of defense of the attacking unit and its position on the battlefield do not affect the amount of damage.

Well, I don't know for sure, but I've observed that the amount of damage from towers also depends on the forces opposing the castle – a kind of balancing act... I have no doubt that there are many other reasons – I know a lot of examples, but I can't calculate them accurately...