It's just that they seem to be too close to the cultists. Don't they know that the cultists hang out there?
If these are Kalevrites, then there will be no benefit from them in any case, since we already have Ogres. If these are not Kalevrites, then there will be a battle. The potential benefit from interacting with them is minimal, so it's not a great option.