I hired six heroes and placed them in front of the castle, blocking the enemy's path. It's one thing to move along a road, and another to attack (even if it costs a turn). After a few attempts, the enemy didn't even reach the city gates!
The next day, I bought reinforcements and easily defeated them.