Posts from Tanta Simulator

TL DR: You can see the progress here: https://ihromant.github.io

SMOrc
SMOrc SMOrc
SMOrc SMOrc SMOrc
SMOrc SMOrc
SMOrc

Hello, dear chat. I think everyone has heard about AlphaZero, which is now considered unbeatable in chess. Under the hood, AlphaZero has something called machine learning on neural networks. And with its help, thanks to a simple training strategy (the machine learns to play the same way people do - by playing and gaining experience), it eventually beat Stockfish. That was a digression.

In fact, there is nothing complicated in the AlphaZero code from the point of view of implementation. (for example, you can see the pseudocode here https://gist.github.com/erenon/cb42f6656e5e04e854e6f44a7ac54023). The question is in the infrastructure (Google released something called a TPU chip for this) and in the number of computers that calculate this model. Also, from the pseudocode, you can see that by replacing just a few parts of the code, you can apply the same algorithm to Go or Shogi.

What am I getting at? I'm getting at the fact that whoever understands how AlphaZero works can (conquer the universe) write AI for any game they want (like, for example, they write bots for Dota that are already beating the champions of The International https://openai.com/five/ ).

Of course, Heroes has something in common with Dota (randomness, incomplete information) and with chess (complete turn-based gameplay). But in reality, in some cases, Heroes is very simple in terms of options. Here is one of the options I would like to consider.

Suppose you have a Temple. Articles 1-2-5-8, mana 120. And 350 skeletons. And a Dragon Utopia is looming. Question for the experts in the chat: should I go there or not? Sanek, for example, doesn't make a move, creates a map in the editor, and runs this battle there 3-4 times, and after that decides what to do at the moment when the opponent curses, "why isn't he making a move?". Leaving aside the jokes, in the Dragon Utopia, everything is very predetermined (dragons will attack certain stacks, there is no morale, we have very few viable options as a response - shield, defense, attack, wait, animate), and so on until victory (or defeat).

In general, the model fits very well with the algorithm called Reinforced Learning. The principle is simple - we go into the Dragon Utopia, make some assumption, get (moderator removed), draw conclusions, try something new. Or we fought well, we think about how to spend less mana.

Now about the connection with another game. There is a game called "tic-tac-toe". I think someone has already played it in pairs or at school. The game is actually very similar to taking a Dragon Utopia with a Temple. There are approximately the same number of options, in the end there is a result, you can vary the number of answers (for example, play not 3x3 3 in a row, but 5x5 or 5x6 4 in a row, or 15x15 5 in a row). If you teach a computer to play this game, then by making simple manipulations, you can teach the computer to take a Dragon Utopia with a Temple.

Sorry for the long introduction, now finally the essence. I was able to teach the computer to play tic-tac-toe 3x3 3 in a row, 5x5 4 in a row so that it does not lose, and if you make a mistake, it wins. I did this using the Reinforced Learning algorithm and machine learning on neural networks. You can see the code here https://github.com/Ihromant/reinforced-learning .

I am almost sure that it can be adapted for the case with a Temple and a Dragon Utopia. That is, teach the computer to take the Dragon Utopia with minimal losses, and if there are no losses, then with minimal mana loss. If you go further, then you can go to taking castles (even to the point of using harpies to siege), it's just more complicated, it takes longer, and you can't skip it.

Now, why did I create all this? Firstly, to brag (well, yes, ego). Secondly, I have approached the stage where someone can help. I see two obvious options so far:
1. I need a model of the interaction between dragons and skeletons. I think that I will be able to integrate this model over time, but firstly, I work, secondly, I have other activities. And this requires diligence and is a bit boring. And training on the wrong model will give incorrect advice. Then the model may grow to the point of taking castles and so on. But I need an expert in computer behavior who can translate his experience of fighting computers into code.
2. Training models is not done instantly. Next, it will be necessary to train step by step. The process is actually quite boring. Started, waited, saved to a file. The file was uploaded somewhere, someone else launched it, trained it, uploaded it. I need a computer with a good video card (I have a laptop, I plan to buy a computer, but not yet). Because to teach the computer to play 5x5 tic-tac-toe, it takes 2 hours of work on my powerful work computer). You can imagine how long it will take to train taking a castle.

In general, that's all for now. Let's discuss.

I'm skeptical about AI, but if it suddenly "takes off," aren't you afraid that it might inadvertently "kill" the online and offline tournament scene? Because the paranoia about opponents hiding an Nvidia box under their desk will only be cured by a direct video stream (and even then, it's not a certainty).

I also used to think about AI in Heroes as a worthy opponent. In my opinion, it's theoretically feasible, but a Herculean task. Even for global, strategic AI. But it would require an enormous amount of man-hours... As for the battles, the core should be a simple enumeration of all possible options for 2 to n moves ahead, based on some kind of tactical "skeleton," with a comparison based on a formula that calculates the ratio of enemy losses to one's own. Of course, before that, you need to analytically derive a real combat algorithm for the AI (dragons) and then you can train a neural network really quickly. The default AI calculates how to inflict the most damage, but only for the current turn, and what would it cost to give it a little more processing time? Three moves ahead, just off the top of my head, is 200x200x200 options – that's peanuts for today's hardware!

aren't you afraid that you might accidentally "kill" the online game in this way?

If you implement the product as new patches and calculators, then yes. But ideally, you should create a bot. And force hundreds of clones of such bots to self-learn in continuous virtual battles.

I am skeptical about AI, but if it suddenly “takes off,” aren’t you afraid that this will accidentally “kill” the online game and offline tournaments? Because paranoia about an opponent having a hidden Nvidia box under the table can only be cured by direct video streaming (and even then, not necessarily).

No, I don’t think so. Firstly, I don’t plan to create a full bot. I already wrote about computational complexity and other things. The plans include creating battles in Dragon Utopia, ideally also capturing objects (because the battle ends quickly there), taking into account stats and magic from the hero's book. Regarding micromanaging neutrals, I’m not sure if I have enough processing power and skill for that. Therefore, the bonus is just okay, actually. It seems like running "Taking of Tombs Complete Edition" on YouTube during gameplay. There are fewer losses (I don’t think it will defeat a super-skilled player), but the timer is set.

I was also thinking about AI in Heroes as a worthy opponent at one time. IMHO, theoretically feasible, but a titanic task. Even regarding global, strategic AI. But many man-hours would be needed...

For combat, the basis should be a simple enumeration of all possible options 2..n moves ahead, based on some kind of tactical “skeleton,” with comparison according to some formula of the ratio of enemy losses to own.
Of course, before that, you need to analytically derive a real combat algorithm for AI (dragons) and then you can train the neural network really fast.

The default AI calculates how to inflict the greatest damage, but only for the current turn, and what does it cost to give it a little more processor time? Three moves ahead, roughly speaking, this is 200x200x200 options - “peanuts” for today's “hardware!”

If you implement the product in the form of new crutches-calculators, then yes. But ideally, create a bot. And force hundreds of clones of such bots to self-learn in continuous virtual battles...

Yes, the main difficulty is deriving a real combat AI. Because machine learning learns from what you teach it. If you taught it about cats, it will also know about cats, but when it sees a person, it gets confused. It can reuse something, of course, but for example, if you taught it to capture Dragon Utopia without Bloodlust, and then with Bloodlust, then you’ll most likely have to retrain it from scratch because it simply won’t know what to do with Bloodlust (the model there is very important).

There was a problem. In 300,000 trials for a 5x5 tic-tac-toe game, the bot would reach a local minimum and then get stuck in an infinite loop. I added logging and ran it for a million trials. It turned out that it does learn, and it learns very well. It turned into an interesting evening meditation. You can run it and meditate on the console, watching how the machine learns.

And what about the AI, have the bots already learned to play without making mistakes?

So, have the AI bots learned to play without errors yet?

5x6 (theoretically, it's a winning strategy). It looks okay, but I'm not satisfied. I'll try to fix it at work tomorrow. Once I get what I want (a flawless game and good variation selection), I'll start saving AI versions.

Update (in case anyone is interested). 1. I reworked a lot of things because initially, I designed everything around Tic-Tac-Toe (for example, I was evaluating based on the consequence of an action, rather than the state-action combination). I also extracted a good chunk of code into the "agent," and now there are many agents, interesting and diverse ones (some controlled by the computer, some controlled by Minimax, and some controlled by a QLearningAI based on either a lookup table, a map, or a neural network). 2. I separated "training" and "gameplay." Now, you can train gradually, saving intermediate results to disk. Accordingly, if something goes wrong, you can simply load the previous result and start again. You can also control how much to train in the current session. 3. There was a problem where the neural network would eventually degenerate into NaN (basically, it would become irreparably corrupted), so I changed the activation function from ReLU to Sigmoid in the first layer. It learns a bit slower, but it's already reached 4 million iterations, and it's still working, with no dependencies. 4. I extracted the game result, but I haven't yet implemented the conversion of the result directly into the neural network's output, but there are no obstacles to doing so. 5. I have an idea to try a convolutional layer, but it would be helpful for Tic-Tac-Toe, but not for Heroes (although if I further complicate the neural network's structure, I could pass the field analysis through it, and the analysis of the rest – magic, hero stats, etc. – through a dense layer). In short, that's what I've been working on. I'll be presenting my progress on Tic-Tac-Toe at my company on September 25th. Before that, I'll polish what I have and create a presentation. If anyone wants to, they can start implementing the state for Heroes during this time (I'm enticing you :D). If not, I'll probably start working on Heroes myself in October. 3x3 learns in 1.5 million iterations to the point where it doesn't lose to Minimax (i.e., it plays perfectly). Below are some images to grab your attention (showing how 5x5 and 4-in-a-row are learned).

Can you imagine how much time it would take to train taking control.

In general, that's how it looks for now. We're discussing it.

Maybe it would be better to make the heroes more linear, like in chess; build a guild and get exactly that spell, 3 different buildings for control spells; remove all Sanctuary and spell scrolls from the map; instead, add something else; regarding AI, there is already a PvP option, we just need to improve it; and we can also create an OH (Online Hotseat) mode for 5 or more players for beginners, so that more players join.

We need to rework the PvP battle option so that a player playing against AI can refuse the battle if it's 100% unwinnable, so they don't have to click unnecessarily.

Maybe we should make Heroes more linear, like chess; built the guild and got exactly that spell, 3 different buildings for control spells; we remove all scrolls and shrines from the map; instead we add something else; regarding the AI, there's the PvP option, it just needs to be refined; also, we could create a newbie-friendly OH for 5 or more players; so that more players come.

We need to rework the PvP battle option with the possibility for the AI player to decline a fight if the battle is 100% lost; to avoid unnecessary clicking.

Oh, I finally found out who the legendary DaU everyone's talking about is.

Make Heroes even more linear than in HotA, or on Jebus Cross?
Interesting.

Newbies won't come to Heroes, at least not super-motivated ones, due to the lack of competitive organizations and high prize pools.

There's a typical example from recent games:
In early 2019, a custom game came out in Dota, Dota Auto Chess, which generated unreal hype at the time, and DAC was watched more than Dota itself.
GabeN realized there was money and hype involved, Riot realized there was money and hype involved, EpicGames realized there was money and hype involved.
At the same time, Drodo - the authors of this custom game - were developing a mobile game and the custom game itself.
As a result, Drodo signs a contract with EpicGames, while Riot and Valve decide to create their own clones.
Since a mobile game can be made without much trouble on Unity, with already existing models, and Riot has its own platform in the form of LoL, these games release on the same day, beating the EpicStore by several months.
GabeN and Riot say, here are our beta versions of the games, come to us, we'll update them and hold tourneys with prizes. And players, like sheep, go to the unfinished crap because money, money, money.
EpicGames realizes the heat is on, and announces a tournament for $1kk.
Then they release their game, and it's rawer than boards that have lain in the rain for 7 days, half the functionality doesn't work and it smells like a failure.
But we announced the tourney, and all the dates are already set.
As a result, the tourney qualifiers are moved to the mobile version of Drodo Auto Chess.
And players, like zombies, go play it, practice, because of the millllllllllliooooooooooooon.
And there are only 2 slots, and a ton of players, and the game is random.
As a result, having played the qualifiers for the tournament, everyone drops the game.
So, what do we have:
GabeN's crap (I can't call Underlords anything else)
Riot's slightly different casual stuff, with a stolen idea and mediocre implementation
Drodo's mobile game lagging 2 months behind in development
EpicGames' unfinished game
And the most beautiful custom game, full of innovations and interesting strats
And everyone has already lost their playerbase...

The bot is taking the Necropolis. If you want to see the result, watch the last half hour. If you want to see all the battles, watch from the beginning.
https://www.twitch.tv/videos/615773537

Do you understand where this is going, I hope? Kappa

Ivan, you are clearly insane, and you need to seek treatment as soon as possible!

I hope you understand where all this is going, right? Kappa

Graveyards! And then factories!!!! :smile54:

Graveyards! And then workshops!!!! :smile54:

and as the culmination of the revolution, guides for the Warlock:smile14: