Uninvited guests.

Could you provide more details about which faction and level you were at when the opponents arrived? How quickly did you clear the territories before the castle? Did you have time to build defenses? Do you have any mods for Mega AI, EWA, etc., that could have affected the difficulty?

Regarding the dwellings, I can take a look tomorrow, but I haven't noticed this problem before when playing as other factions. Perhaps you have a mod in Maps that affects the dwellings?

Added after 1 minute
P.S. For me, the most difficult starts were with Haven and Fortress, and the easiest were with Sylvan and Dungeon.