Precisely because I've played so many games recently with little activity, I can, with a clear conscience and absolute certainty, state that this isn't how it works. I was eliminated for this reason, both as a Mafia member and as a Town member. It works both ways; it's a matter of luck.
You're currently looking at it from your perspective. And in the only Mafia game I've played on Hv, I accused you of being Mafia due to your inactivity, and it turned out I was right. Isn't it logical that after that, all such inactive players seem like Mafia to me?
Added 2 minutes later
Also, in my Mafia game, forgotten ghost was mostly inactive, and heroist started talking when the pressure was on him. I have every reason, based on this experience, to suspect that inactive players are Mafia; don't you think so?