It's quite feasible. You just need to avoid encountering the dwarf priests from Grimheim (or wherever they come from), who only appear after you've captured all the cities of their brethren/elves on the map. In this case, you'll receive a message suggesting that it's best to avoid this hero. However, you can and even should eliminate the other dwarves. Personally, I didn't have any problems with this.
I understand that this is a secondary task – you can do it if you want, or not. But I think the more realistically achievable tasks there are, the more interesting the game becomes.
But this is just... a thought