I've fought Ifrits countless times, and I've noticed that as the attacking party gets smaller, the effectiveness of their fire shield (Winds of War) decreases.
Yep, that's how it is – but not because there are fewer Ifrits, but because the attacking party itself is smaller. Consequently, the damage dealt to the Ifrits is less, and the cooldown, which depends on that damage, is also reduced.
It's quite simple logic – what surprised you?;)