Building an AI that is fun to play against.
Creating an artificial intelligence for a game has always been a daunting task. Players don’t immediately see when an AI is doing okay, but they will certainly do when it’s doing bad. Here is how we have been approaching the task of making an AI that does not act terribly stupid and is able to play together with a human player in a game that is build towards 2v2 matches, without cheating… too much.
There have been some great advancements in gaming AI recently. The most prominent example is probably the Google DeepMind project, which just managed to beat some of the best Starcraft II players. Though deep learning is an awesome technique to build a powerful AI, it also has tough requirements, e.g., a lot of machine power to simulate enough matches and data, loads, and loads of data.
Our approach for making an artificial player is more traditional; we will build everything we think an AI player needs manually and tweak it until it feels good.
Micro and Macro AI
We actually have more than one kind of AI in our game. The first one is called character AI and handles single units on a micro-management level. For example, moving a unit to a target location while scanning its surroundings and check if an opponent unit is in range. If a unit is found, move towards it and attack. Most of the character AI is built using Unreal behavior trees, which are a great solution for this.
The second one is the player AI. It will drive the high-level strategy of the player and do everything it can to beat all opponents and win the game. For the rest of this article, we will only focus on player AI, since the character AI is a whole other beast to talk about.
Making a plan
Behavior trees are the built-in solution for AI-related tasks in Unreal, and since it worked great for our character AI, the first iteration of the player AI was built mostly in behavior trees. However, it turned out that, even though behavior trees are great to make individual units act smart, they are not well suited for long-term strategic decisions like: “If our current mine runs dry, try to find a new one. If the new one is guarded by a creep camp, send the army to clear it first. Then collect enough resources and send a builder unit to build a new main base building at the right spot near the mine”.
So it looks like we need a system that allows us to define a set of high-level tasks like “build me a unit” or “build an expansion there”. Good thing there already is something like this; it is called a planning system. Planning or goal-oriented design has been around in game AI for a while, and the main idea is to have sets of goals and actions. In short, goals define something we want to achieve, and actions change the state of the world and are locked by preconditions.
A simple example of an action from A Year Of Rain is to train a unit; Training a unit only has two conditions: (I) Owning the building(s) necessary to train the unit and (II) having enough resources.
The system that is responsible for training units could now have the plan of having 5 worker units. To achieve this, we will create 5 „TrainUnit“ goals and add them to our planning manager queue. When all conditions of any of the queued actions are met, it will execute its effect. In this case, actually training the worker unit. We will use this system to define the very basic behavior of our AI. On top of that, we need more systems to decide what our actual goals are and make sure that we can achieve them. To do that, let’s go on and break down what a match usually looks like:
- Select a player role.
- Start by training a couple of builders and assign them to collect resources.
- Depending on our play style and overall plan, continue training builders, units and construct more buildings.
- Send a scout unit to find out where our opponents are and what they are building.
- When the hero spawns, either immediately go and farm XP by clearing creep camps or wait until we have enough units if the hero is not strong enough.
- Level up the skills of our hero.
- Research tech that matches our selected player role.
- Either find or know the other important points of interested on the map, e.g., healing wells or watchtowers.
- If the mine is close to running out of resources, find the best spot for an expansion and built one.
- When our hero and army are strong enough, find the weakest player in the opponent team and attack.
- While fighting with the opponent teams, remember which units they are using and prepare to build counters.
- Also, do not forget that we have a team partner! Make sure to keep an eye on what he is doing and support him in fighting creeps or players.
Managing the mess
Wow, that is quite a lot to do! Even for a human player. Let us try to break that down and see if we can find some categories to make it more manageable.
Our general game plan, i.e., what buildings and units we want and how we want to skill our hero, is usually known before we even start playing. Players of RTS games will already know the term for this; it is called a build order (BO). Having build orders allows us to define different types of AI personalities just by telling them what to build and research. For example, we could create an AI that just constructs base military buildings and trains nothing else but cheap units to rush opponents. Or one that ignores early units and aims for a fast technology advancement. To allow the entire development team to create new build orders, we decided to create custom assets that can easily be created and modified.
In addition to that, we will need a system that manages our army. This system is solely responsible for choosing an attack target from all possible targets provided by other systems. It will also keep track of the observed or estimated strength of the target of an attack and, if necessary, request other systems to train more units until our army is strong enough. This also includes periodic checks during an attack to decide if we want to retreat, if too many units, or our hero, have died. To be able to manage all of this, we need to able to at least have a coarse separation between low and high importance targets. For example, if our base, or the one of our team partner, gets attacked, we should probably go and defend it instead of moving to the next creep camp.
A surprisingly difficult task for the unit manager is to find a good spot near our main base to gather units. Since the army can consist of quite many units and space is limited, larger groups of units tend to block workers from reaching the main building or block the space around buildings that finished training a unit but now cannot find a spot to put them, so they can‘t finish their production. Additionally, since our map layouts differ quite drastically from map to map, it is hard to find a general direction from the base that we can safely put all units. There are a couple of possible solutions to this problem, e.g., simply selecting a random point and hope everything works out or shifting the task to level designers by letting them set a rally point. For now, we are going with the first option with some distance to the main building. That seems to work fine, even though it sometimes blocks space for a while.
The third system is all about resources. It will manage the workers and assign tasks to them. Even though the build order generally decides how many builders we want to train, we still need to make sure they are working all the time. When one of our buildings is damaged, this system will also manage the amount of workers assigned to repairing them. Additionally, the resource manager will keep track of all of our mines and, if the resources are running low, initiate the construction of a new expansion, though it does not care how and where. That will be taken care of by other systems.
The next thing we need is a system that will keep track of what we currently know about the state of the map. This includes knowledge gained by analyzing the static data of the map, so everything that our level designers have placed on a map, including creep camps, expansions, and possible player start positions. Even though using this knowledge could be counted as cheating, because it also includes the difficulty of the creep camps as well as knowing which expansions have creep camps guarding them, the player also has access to this information. Either by having played this map before or having a look at the minimap.
In addition to the static data, we also need to keep track of everything that happens during the game. Of course, we could do this by simply abusing the fact that we are very close friends with the memory. We could just constantly iterate through all units of all players and see what they are doing. But that would not be very fun, would it? So instead we will do what the player will (or should) do, too: Build an early scout unit and send it around the map to points where we know that there could be player bases and remember everything we see on our way. Afterwards, we need to constantly evaluate what we see on the map and remember it, so we can later decide when to attack our opponents and if we want to build counter units. Checking every unit’s vision in a regular interval is obviously a big performance hit, so we decided to select a batch of random units per update interval to keep this in acceptable limits.
The last thing we need is a system that puts everything together, one that drives the strategic decisions, instructs the other systems when to do what. We will call this our strategy manager. This might sound like an interesting system, but to be honest, it is only a collection of data, timers and event listeners that talks with the rest of the game and forwards necessary information to the other systems.
Talk to me!
A Year Of Rain has a big focus on 2on2 matches, so an important aspect for a competitive AI is that it needs to be able to communicate with a team partner, no matter if that partner is a human player or an AI. Our game already contains a couple of features for communication, most importantly a text chat and a minimap ping system. Since we strive to make our AI feel as natural as possible, it makes sense to use those systems for communication.
Our most used feature are the minimap pings. We are using this in both directions; if we are attacking creeps or a player, or our base is attacked, we will send a ping to that location. At the same time, we are always listening to pings from our team partner and support him. This creates an interesting dynamic when two AIs are playing together, where usually the slower player follows the faster one, creating one large army roaming the map in the end. When playing with an AI team partner, a human player can also use this feature to make the AI move to spots on the map or stop an attack that he thinks would not be successful. This helps a lot in creating an AI that feels smarter than it actually is. In addition to that, the AI will also send text messages when playing with a human partner. This is usually just a hint, e.g., for the next target of an attack, but this also opens up possibilities for a guiding system that would send helpful information to newer players.
Making it scalable
One thing we did not talk about at all yet is difficulty. Up until now, we tried to make the AI as smart as possible to be able to win games. However, if we succeed in doing so, we might end up with an AI that is way too hard for beginners, and that’s also not very fun. Players should be able to start playing against an AI, get used to the game, and improve their skills to be prepared for online matches against humans. So we need to make our system scalable to support a wide range of players from beginners to pros (… well, probably not pros).
Fortunately, we have many things we can tweak.
To name just a couple of examples, we can increase the time that passes between training units or delay sending our hero to farm XP. We can add a factor to our evaluation if we are strong enough to fight an opponent, so we are making wrong decisions and wipe our group. However, since we could be playing as a team partner with a new player, we also have to keep in mind to not make weirdly wrong decisions and confuse new players.
The Future
There are still a lot of tweaking and adjustments to make before I’m really satisfied with the AI behavior, e.g., we still have some issues finding expansions to eliminate a player and some minor f*ups caused by the complexity of the system and the interconnection between different tasks. But thanks to the goal-oriented approach and the clear separation of responsibilities it is only a matter of time… as it is so often.
Michael Matzen
Michael Matzen is a console/generalist programmer at Daedalic Entertainment. Michael is working in the game industry for about nine years now, four of them at Daedalic. He studied computer science at the University of Kiel and promoted in 2010. Always eager to learn and try out new stuff, he has brought games to a wide range of platforms before joining A Year Of Rain.