How to change the damage formula for spells so that it depends on the hero's level.

Can I change the formulas in the game engine? And where is the engine file located?

The engine is the "exe" file; I've never changed it because it's program code. It's located in the "bin" folder within the game's directory, and that's what you use to launch the game.

Can I change the formulas in the game engine?

You can change them, but honestly, I haven't kept up with that area of modding. I saw in a card-making chat how one guy used some kind of patcher to increase the number of artifacts in the engine... I don't know how it works.

And where is the engine file located? Is DefaultStats an xdb file?

\GameMechanics\RPGStats, but don't take it from data.pak; take it from a2p1-data.pak.
There's a lot of stuff in there. Of the things I've changed, I can remember modifying starting resources, numerical values for some creature, hero, and specialization abilities, undead raising tables, blood rage bonuses, bank protection, and a lot of other things that I've never touched