[2x2 Tournament] Double Up [SoD]

Since the topic of scrolls has come up, here's an interesting question, not just for this tournament, but for the rules in general: Let's say there are 2 Barbarians, and I build Towers in both and legally use scrolls. The next day, I capture a Castle. Do my Towers in the Barbarian faction retain their original strength, or do I now have to build a Grand Mage 4 in the Castle?