Well, I wouldn't say that. It's unlikely that these are the weakest Castles; rather, according to statistics, few people play as them, so it's difficult to judge their strength.
And while the Stronghold's tactics are indeed quite diverse and depend on many parameters (including luck), I wouldn't dismiss the Swamp. :)
Specialization in Defense, in my opinion, gives the army tremendous resilience. For example, Tazar, upgraded to level 10, reduces the damage dealt to the Hero's troops by 22-23 percent. And with a favorable combination of circumstances (for example, expert Earth magic - casting such a simple spell as Shield), the damage is reduced by another 30%, i.e., in total, it amounts to more than half.
And what's the problem with building a Grand Mage in one of the captured Castles?
2. The problem is that you first need to capture a Castle. When playing against the AI, this may not be a big problem (although anything can happen), but when playing against people, it is quite difficult to do without good magic, which the Swamp Castle does not have as much as the Stronghold.