Snakes are certainly a pain, but it's usually difficult to gather enough of them because their stats are low, making them an easily defeated unit. I agree about Basilisks and Gargoyles (especially Gargoyles). Wyverns are very weak on their own (if I'm not mistaken, they are the weakest level 6 unit). Hydras will be mostly dead by the time they reach the battlefield, and they are easily dealt with by creatures with a double attack (e.g., a Crusader). And that's all there is to the Swamp.
Hydras + a Swamp hero = a death machine, slow but invulnerable. Add Earth magic, which Witches use, and the Hydra turns into a mega-monster, and no horde of Crusaders with a bunch of effects on them (Snakes weaken them and make them even weaker) will stop the Hydras.
You forgot Gnolls and Lizards. I agree that Lizards have great stats for their level, especially defense. So, Lizards will last a long time and will be able to outshoot enemy archers.
I already talked about Gnolls.
In my opinion, the only Castle that can counter the Swamp is the Barbarian Castle with its monster that reduces 80% of defense (all the Swamp's efforts are for nothing). Otherwise, the Swamp is quite strong.