It seems that the golem is a monster that is worth fighting only after you have leveled up. I draw this conclusion based on the fact that it moves very slowly. It won't disappear when you progress to the next acts, and by the end of the game, it will likely reach the village. I think it's worth going after it later.