How exactly does this manifest in your case? Did the hero change color, or did he wander into your tavern?
Here's the situation:
Two allied castles are next to each other. One (red) is inside a castle, specifically within the castle itself (so it's not visible on the map). The other (blue) comes to visit him to learn magic (or replenish mana, it doesn't matter). When you press the spacebar, the heroes switch places, but the blue hero loses his hero. That is, on the next turn, the blue hero's hero will be added to the red hero, but with the red flag... That's what happens.
The question remains.