You can lose one there too, if you take the shield before the battle with them, but that doesn't fit into a quick playthrough, so I resigned myself to losing two genies.
You suggested it, so you should explain it, and I'll support you...
And as for explaining - I built the tower not in my own castle! :)
Added 1 minute ago
My second, third, and fourth attempts were 15 days.