I studied this issue in detail in the appendix to the second Heroes game, and I also studied materials written by others, and came to the conclusion that it is impossible to create a formula.
Here, we'll need to create two lists: one taking into account the increase, and one without it.
Regarding special buildings, there are also many questions: for example, I built the Fairy Glen very often, but I never built the Skeleton Mound. And I've never seen players do this in actual gameplay.