Possible rules correction

(because the situation with the fortress on the hill, isn't it?).

No. The fortress can be found anywhere and upgraded (some levels are even free), but this requires you to come to it, and there's a risk of getting it, rather than getting it for free.

The difference, in my opinion, is noticeable.

And the fact that you weren't there is your problem. The same can be said about new players - newbies - who play with 1-2 heroes, like "I was far from the castle, and you suddenly came and took the main city, I didn't have time to run, that's unfair." Even if you had other reasons to retreat. But your city was attacked without reinforcements, and the fact that you couldn't defend it is your problem.