1. How many weeks in a row you own the castle (it increases with each new week).
2. The more defenses you withstand, the more damage it does... and + the amount of HP of the enemies defeated during the defense...
P.S. Basically, towers gain some kind of experience... personally, I managed to get the central tower to deal 300-350 damage.