I've cleared everything I could... so I thought, maybe I just took too long. If I attack the target Castle directly from the central Castle, will the enemy's army be smaller? Or not?

In some missions, the enemy's army doesn't grow, but in most, it grows over time. I don't think it makes any difference when you complete the mission. After all, the player's army also grows, artifacts are collected, and the hero levels up.

Exception: the 5th mission for Markel when playing as the hero.

If you want to set a speedrunning record, you need to level up logistics and defeat the enemy as quickly as possible. If you want to make the next mission easier, you need to max out the hero, visit all the stat-boosting locations, and collect a full set of spells.