Yes, everything is fair when it comes to you. You need to behave normally, not depending on your mood.
The problem is not minimal at all, considering that there are many moderators, it is impossible to keep track of everyone, and many also have a factor of bias when making decisions about awarding points. ;)
Oh yes, can mentioning "inadequate" and "abnormal" people in previous messages be considered an indirect insult to people who have already received points?