How do I edit the game engine?

In a previous question I asked in this section, while trying to figure out how to change the damage formula for spells, I realized that this would have to be done by modifying the game engine. Can you explain how to open it for editing and what it consists of (perhaps Lua code or something else)? I have a lot of experience in programming, but I haven't delved into this area yet.