How do I properly move buildings within a town?

Did you modify buildings in GameMechanics/TownBuildingSharedStats?
then in the building files you look for the line:

<UIObjectName>Dwelling_3</UIObjectName>

and change it to the desired one (say, to swap the minotaur dwelling for the witch one, instead of Dwelling_3 make it Dwelling_6)

and in general, it's better to change town maps via UI/TownBuildNew/Towns/, where files like town definition are located, which determine the placement of buildings and arrows on the town map (the building hierarchy, however, still depends on the TownBuildingSharedStats files)