I agree, but how can you break through to the portal in the first week when you have so few troops?
I would gather skeletal archers and raise ghosts, then use them to take down more valuable units one by one.
Be careful with your Lord, otherwise you'll see what happens.