We are clearing out the neutral creatures.

Yeah, dealing with neutral creatures at the beginning is a problem. It's not hard to capture mines, but then I don't know what to do. It seems like you need to develop as quickly as possible, but there are no troops, and it takes a long time to accumulate them due to the growth rate (I mainly play as Dungeon). And while there are few troops, there's a chance of losing the Furies, which are worth their weight in gold. And even when you have a decent army, I still try to be as careful as possible with neutral creatures, which is why my hero isn't very developed, but the army is almost equal to the total growth during that time, which is worse, I don't know, maybe you need to find a compromise.
But Haven archers, like most other creatures, don't cause any problems at all – Stalkers are 100% effective against them, and that's a huge advantage for Dungeon. But there are problems with the new Elven archers; sometimes they move before the Stalkers, even though they have lower initiative. In a game against the AI, I save before attacking them, and in online games, I probably just won't attack if they're not guarding anything particularly valuable.