I wouldn't take Chaos mages at all. Their troops have decent magic resistance, and they also have mini-artifacts that reduce magic damage. You should focus on attack to eliminate troops quickly. It's better to take attack, light, luck, and leadership (for the aura).
I'm not saying you need to level up Chaos, I'm saying you can take Magic Weakness through Chaos to weaken the mage's damage!!!