The Temple of the Sea only generates on water, which is currently not playable. If there are attempts (and there will definitely be some someday) to make water playable, then it will be fixed, of course.
The Ivory Tower only generates on sand and snow, and with a very low frequency and inconvenient value. I've played with five different factions, and only one had even a single tower. In one test generation, there were actually 4 in one treasure, but that's just luck.
Let it generate everywhere, like the Hive, just adjust the contents.