A topic was raised on Steam about capturing empty castles. It's a real absurdity that there isn't a guard (growing with the city's level) to protect against such an attack, when you're far away, and an opponent with a few weak units takes an empty castle.