Which castle is the most difficult?

When playing against the AI, the choice of city doesn't really matter. You're right. But when playing against other players, it's important not just to "click on everything with the right mouse button," but also to maintain a good pace. The outcome of the game depends on the speed of your development, and it's much easier to achieve this with some cities than with others. That's the whole point.