You're not reading carefully. The Fortress guards the entrance to the Dungeon. On the first week, you don't even need to bother with them...
After the rearrangement (or castle placement), it became easier - and I passed it right away.
All I had to do was freeze the hero with the Elves on the first week.