The first player chooses one of the castles. Then the second player.
Therefore, 2 castles are inactive.
You can leave 3 castles... or 5... or even all 8... but I personally settled on 4, if you don't like it, suggest how many can be left.
I'm for the 2nd option.