Red crystal

I'm playing the "easy" version with a human hero (I failed towards the end of the second month in the hard version). So, it turns out to be a mess. After repelling several sieges, around the middle of the second month, a bandit appears and offers to block the portals for a month in exchange for ownership of the castle for a week. If you refuse, an enemy with a huge army appears two days away from the castle, which is impossible to resist. As a result, you agree, but the castle is not returned to you after a week, or even two. For two weeks, I completed tasks (portal, clearing the path, caravan) and even captured a castle with a shipyard. But here's the problem, another enemy with a huge army appeared again, even though only two weeks have passed since I met the bandit. Is this some kind of deception – the supposedly returned castle after a week, a month of peace – or is it bugs? And also, is there too much "meat" (units) in the enemies' armies?

In general, it shouldn't be exactly like that. The bandit offers to take the castle for at least a week, i.e., in any case, at least a week will pass before you can talk about its return, and at most for a month.
And the enemies should not appear while the castle is under control, and the path to it should be blocked for everyone. If an enemy appeared while the castle was under the control of the bandits, then something went wrong, some kind of error.

Added after 32 minutes

And also, is there too much "meat" (units) in the enemies' armies?

In general, I planned it so that at any given time, the enemy could put up a decent resistance to the player. On his side (on the next hero) is a large army, fully equipped with artifacts and spells of the corresponding schools of magic, and on your side are strong castle walls, many stat-boosting locations on the map, and non-AI elements.)
2 castles with all creatures built and captured dwellings will help prevent the army from weakening.

I have some ideas on how to further weaken the enemies, but the main thing is that it doesn't turn into a massacre, starting from a certain point.