You can't remove the arrow from the castle to the Knights, as this will lead to overly powerful builds based on Knights and Angels. Neither of them needs the castle; you can build something more powerful instead.
Griffons are safer in this regard. But if you remove arrows from the castle here, it will lead to a situation where, when the city reaches level 6, Griffons will be built without a castle.
Two arrows from the fortress-castle-stronghold (and in principle for any other building) are also bad because they make building it too much of a priority over other buildings (two arrows mean minus two options for building!).