After Rutger and Elia, only minor units were running around my castles.
Here's another thing, I'm not sure, but if you play as fast as possible, like in the walkthrough descriptions (as I did the second time around), Rutger appears with a very weak army, and only when the turquoise player has one castle left. And not like it was the first time, when he showed up while we were rebuilding Gryphonhill.