Without magic, could you use ranged attacks to run away from them?
The problem of "too easy" neutral units to defeat is easily solved by introducing a large number of mixed units. And the few light neutral units that remain will be in the minority and will not have a significant impact on the gameplay.