I suggest adding a mandatory restart to the rules if monsters of level 2 or higher guard the exit from the castle or one of the mines (wood, ore).
Then, only the restarts would take at least 2 hours. In principle, as much as I've played, 2 restarts are quite enough. You can use them when such a situation arises.