I am AGAINST taking points away. TRY TO CONVINCE ME.
Example 1:
A player who plays weakly approaches one of the leaders and says, "Let's record a couple of games in your favor. This way, you will solidify your position as a leader, and I will climb higher in the ranking."
According to your system, such manipulation of the ranking is possible, and you couldn't come up with anything worse.
Regarding the judge – the moderators probably already have a lot of problems, and I don't think it will be so malicious.
Let's say we have our first game, and you win. After analyzing the saves, I tell you that you used cheats (again, this is just an example) and I don't confirm the game. What should we do in this case? We will shout at each other, use all sorts of foul language, each insisting on our own, until the moderators close the topic or something else happens. Before undertaking such serious events, you need to think about it many times and look a little ahead. No tournament ever happens without some controversial moments, and you want several dozen players to calmly play and enjoy life? I would be very happy if there were never any quarrels, etc., but this is life, in which you need to be prepared for anything. You don't want this venture to fail due to shortcomings in the organization, do you?