Is the third one the farthest from the town? Are all three peasants active when the quest to collect taxes is taken and Harry is visited? One of the peasants joins, one pays with resources, and another gives a quest about bandits. As soon as the bandits are defeated, he moves away from the path and becomes passive
Yes, it's the farthest one. One joined, the second gave two crystals, after which it says: "Quest completed." Visit Harry, and after visiting, two farms join, and the third one sends a quest; there was no quest about bandits.