Rollback between castles during auctions.

Regarding roads, I don't play on fast maps, but I have the impression that it's quite easy to gain an advantage on them. So, it's worth supporting a good castle.
As for the opponent, this aspect is more noticeable when playing against experienced players, because, as you correctly pointed out, many of them are well-trained and won't allow you to gain an excessive advantage, even with relatively equal castles like Tower-Castle.