I didn't quite understand what you meant by "tied to the castle"?
I meant tied to dwellings, I misspoke... And sometimes to the castle as well. You constantly need to manage the growth, hang around each dwelling, returning to it every time the growth occurs. Eight heroes with 2500 units each – that's noticeable, and instead of developing, you can undermine the economy.