I see where I made a mistake. I mistook the cave in the cliff under the Kamisato estate for the corresponding temple itself, but it turned out to belong to Chinju Forest. I figured this out when I decided to redo the puzzles and got confused by this:
In my memory, this (judging by the pattern) was one of the first and easiest locations in this quest, so it shouldn't have been one of the last three. Overall, it's a good quest — it reminded me of the old games from the era when you actually had to strain your brain. It's just better to do them all in one sitting, or at least not stretch them out over a very long period, or you'll get confused.