The weakest castle in Heroes of Might and Magic IV!

In my opinion, the weakest are Life, because their units are terrible, and the army's mobility is the worst (the archers, squires, and Crusaders (or ballistas) are slow).

You just haven't played against someone who plays Life as humans. Trust me, it's a worthy race. And by the way, it can easily overwhelm Death because what you mentioned requires a lot of time. When playing against Death, the best tactic is to play quickly.