Maybe we shouldn't go through the whirlpool?
I mentioned a bit higher that
you return to the Temple via a different path.
I don't remember exactly, but there's either a descent or a portal to a neighboring room where the sword is located.
or here:
We go through it, and in front of us will be a passage leading up the stairs. This is the exit to the surface, to the Castle of Kaishino (lower left corner). In a mythical way, you can then use this descent and go further. Before that, it was only one-way for me. We go down this descent, continue further, and the sword will be in the center.