The size of the armies increases for everyone; the percentage of this increase depends on the chosen difficulty level (as far as I remember). Therefore, you need to act according to each specific situation: sometimes attack immediately and head-on, and sometimes wait, first capturing nearby neutral forts and castles, thereby further increasing your army. However, delaying the attack for too long is also not a good idea – the opponent is also not idle and is also capturing nearby neutral forts and castles.
If the enemy is clearly stronger than you, it is better to start the attack earlier, seizing the moment when the enemy's city or fort is unoccupied, and capture it there. If it will be possible to gather reinforcements later, it is better to upgrade the city. Otherwise, attack the enemy with small raids.