Which castle is the most difficult?

When playing against the AI, it really doesn't matter which town you choose. You're right. But when playing against other players, it's important not just to "click on everything with the right mouse button," but also to maintain a good pace. The outcome of the game depends on the speed of your development. And it's much easier to do this with some towns than with others. That's the whole point.