1) either rely on speed, so your crossbowmen go first. And ideally, other units too. Then there won't be much resistance.
2) or you don't take crossbowmen. The Castle has griffins and knights for such battles, and angels from the conservatories. Plus, monks shoot well and don't have defense problems.
In my opinion, that's how it is... Also, it should be noted that there are fewer ranged units in the game.