Which version are you playing? In version 1.2, even a stack of goblins can't stop anyone anymore. It seems they've made the ability trigger like it does for trappers in the fifth game. That is, the probability of the ability triggering depends on the ratio of the goblin stack's strength to the neutral stack's strength.
One goblin really doesn't slow anyone down. Except for maybe one Vestal. :)