well you should defend the Castle, even units of 1 in it under towers already deal damage, but otherwise attacking on day 1 is forbidden even with your growth, not just the native one anymore, but why can't you buy it back if he snatched it from you, the proposal is absurd IMHO
it's possible to play by betting on this. I see that the opponent has archangels in the tavern. I have 7 angels. Why not show up at his place when he is, for example, clearing his Dragon Utopia or treasure zone? I'll upgrade the angels and buy up growth from caches as well. And let him return as much as he wants.