Let’s come up with a monster.

Why not? Instead of shooting towers, you could have one monster acting as a defender. And its strength would depend on the castle's level.

If there's a defender monster, then, as you said, the towers need to be removed, which is unlikely. And then... a new building would need to be constructed in the castle for it. Or not?

Added 2 minutes later
Someone mentioned a golem that attacks walls but moves very slowly. That's a good idea.