If there is no other adequate way to ensure that Angels are not built before turn 5, then let it be.
For example, a starting strategy with Stronghold-Archers-Griffins + Angels with an upgrade will be dominant. But if Angels require one more construction turn, then either Archers, or Griffins, or the Angel upgrade will not fit in this combination, and that will balance things out.
If it's not necessary, it's better to remove the connection to the Stables, thereby slightly increasing the number of effective starting strategies with Knights. In secondary cities, the Stables will be built to support a mobility strategy on "their" territory, which gives a certain racial characteristic, as it was in the original game. On the other hand, building them requires raising the city level to 6, which forces you to spend resources and turns. I think it will be better without the connection; if something goes wrong, it can always be rolled back.
But nevertheless, I still strongly doubt that this is the right thing to do, because it weakens the concept of Strongholds as the core of city development. For the same reason, it is desirable to maintain the connection from the Stronghold to the Archers. But if Archers are just a minor thing, then Knights are an elite human unit (there is nothing higher, Angels are not humans).
Developing a settlement from level 3 to 8 is not so difficult even without a Stronghold, but whether the knightly elite, the pinnacle of human military power, should be stationed in such a settlement is very questionable.