Heroes 5.5 WarGame Edition - Tactical Edition.

The point is clear. It's not worth removing a building just for some dubious "improvement."

Nothing will fundamentally change in the build order if you replace this building with a bonus +10% to sworn enemies. With the current scheme, only one of these two buildings has a place, in any case. Out of 30 slots, 7 are for dwellings, 7 for upgrades, 3 for forts, 4 for money, 5 for the guild, and the rest – 4 auxiliary buildings: tavern, forge, market, and racial building.
Although, of course, the +10% effect is extremely weak and will have to be supplemented with some +1 to attack.