Also, if your focus is on speed, I think you're taking Erling for no reason. He might be useful for the final battle, but for breaking through defenses, Ingvar is better. Or maybe Ibba, but she's a little worse in my opinion.
Be more aggressive, take risks. Save time: return to the castle less often. That's about it... Don't neglect logistics if you get the chance.