How do I edit the game engine?

You have extensive programming experience, but you suspect that the game engine is written in .lua? That's commendable! I recommend contacting the user dredknight, if he doesn't respond here; I think he will provide you with links to articles on the fandom wiki about editing the game's executable file.