The main thing is to scout it early and understand what is needed to capture it.
In ConF, the phoenix external dwelling is taken easily. The primary strength there is magic: build level 2 magic or learn Bolt/Lightning and send a mage with six fairies into the same external dwelling — the result of this battle: minus 2-3 phoenixes. Then you can finish them off without losses using ordinary troops and Astral Thoughts.
Or you can send another mage in later — the result: the external dwelling is taken with a loss of only 10 fairies :)