Emm... Don't send a garrison into the castle.
Wait, you brought priests with you, arrived at the city, touched the city, an assassin appeared, you agreed to the assassin's terms, the castle went to the green faction, you were kicked out to a free cell, and now you can't attack the green castle?