Sorry, I got confused... I just hear "Bring back the swamp, get the hydra out of the Dungeon!" so often that I didn't think about it. But you're wrong about the dwarves! What do they have in common with elves and the Dungeon?
1. I hear much more often about the return of the Barbarians than about the swamp.
2. Dwarves are miners, meaning they should live either in Sylvan or in the Dungeon to mine gold.
Find me a reason to put dwarves in any other town.