Oh, your walkthrough turned out to be quite strange :) I didn't fight anyone except neutral creatures and the one near Isabel's castle. It's easy to pass by the orange one; he just "patrols" his territory. While he was walking around somewhere, I even managed to enter the fort on the hill and upgrade everyone. After you defeat the mages, the orange one only goes as far as the dungeon, and then it's the turn of the purple one to "patrol," although he didn't care about me at all, so I just walked right past him, and then it was very easy.