Making of Iron Harvest: Multiplayer & Online Services

Instead of classic server structures, for the Iron Harvest KING Art relies on microservices.

Early in the concept phase of Iron ­Harvest, we knew we could not ­offer a real-time strategy game without real-time multiplayer ­featuring dedicated servers and matchmaking. Our focus was always on developing an ­extensive and great single-player campaign. In addition to that, we wanted to offer a cross-­campaign online “metagame”. This should motivate players to try out all parts of the game with experience points, achievements and regularly changing challenges and offer an incentive to start the game again after finishing the main game. Since we did not want to force our customers to be connected to the Internet for Iron ­Harvest, these meta features were always an optional bonus.

Our first and for a long time only published boxed game with multiplayer was Battle Worlds: Kronos in 2013, a turn-based strategy game. The multiplayer mode was a “play by mail” variant. A player would send their moves to a server and then hand the game off to their opponent, who could then make their moves hours, days or weeks later. This did not require a real-time connection to a server, and even aside from multiplayer, the Battle Worlds servers required very little functionality.

KING Art has also been creating free-to-play browser and mobile games for many years. These games are only playable ­online. For this, large and stable ­servers are used to simulate and validate the gameplay. Unfortunately, adapting these servers to the requirements of Iron Harvest would have been equivalent to the effort of a new development. Nevertheless, the experience and lessons learned were indispensable for the creation of Iron Harvest online services.

So, for Iron Harvest, we were in a situation where we had no existing technologies and very little experience. But this also gave us the opportunity to design and develop a new and modern system from scratch — the dream of every programmer.

The Past: Apache + PHP + MySQL

For Battle Worlds: Kronos we used a PHP backend with a MySQL database. This was common at the time and had the advantage that such a server could be set up quickly and easily with almost any hosting provider. The complete functionality of the server was in a single, central web application. With increasing requirements, this became more and more complex, extensive, and difficult to maintain. In the case of Battle Worlds: Kronos, the scope of the server was manageable, but for larger projects that are developed over a longer period, hundreds of thousands of lines of code and thousands of classes can be created. The server had to be taken offline for even the smallest updates, and errors in parts of the application could lead to complete ­system failure.

The Present: Microservices

If you look at backend solutions in 2021, you will not be able to avoid the term ­“microservices”. The idea behind this is that large, monolithic server applications are avoided. Instead of a single server application that performs all tasks, many small ­applications are developed. These so-called services are packaged in containers (Docker, for example) and deployed to “the cloud”. Each service should have a clear ­responsibility and should only take care of this task and nothing else. For ­example, one service could take care of ­login while another delivers player profiles. There must be clear interfaces between the services through which they communicate with each other.

Designing a system according to this ­approach results in several advantages:

• If the scope of a microservice is well defined, this reduces complexity in development. The services can then be developed and adapted independently of each other. The developer of a service only needs to know the interfaces of the other services, not how they are actually implemented.
• Parts of the online services can be updated independently without putting other areas at risk with an update. For example, a change to the store service cannot inadvertently lead to an error in the login.
• Since each microservice is a separate process in a separate container, the failure of one service does not lead to a total failure of the entire system.
• If it turns out that a specific service is very computationally intensive or is used very heavily, it can be selectively distributed to several machines independently of the rest to share the load.
• Different technologies can be used for different services. Thus, the same ­programming language does not have to be used everywhere but the one that is best for a particular use case.

Of course, microservices also bring some new challenges:

• Microservices consist of many small, moving parts that must communicate well with each other. A supposedly simple process can be influenced by many services. It can be challenging to keep track of them all.
• Since it is possible that different ­versions of services are in use in live operation, care must be taken to ensure that the interfaces between the services have been created conscientiously. That compatibility with previous versions is ensured.
• Careful consideration must be given to whether and how recurring code is shared and updated between services.
• Microservices are designed for clusters of servers. This makes local test servers on development computers impractical. So, care must be taken to ensure that there is a good way to develop and test services early on. At Iron Harvest, we had a test and development cluster early on in addition to the live cluster.

But when is it worthwhile to move to ­microservices? Especially if there is ­already an existing system for a longer period, it can be very tedious to subsequently establish a microservice architecture. Also, the effort to run a serving cluster should not be underestimated. Since we had a fresh start with Iron Harvest and thanks to our publisher Deep Silver, we had an excellent ­devops team at our side, the question of whether we wanted to use microservices was a clear yes. The advantage of an ­excellent scalable and well-protected server solution ­outweighed the disadvantages for us by far.

Powered by Kubernetes

The foundation of Iron Harvest’s online services is Kubernetes, an open-source system designed by Google to deploy, scale, and manage container applications. Applications that are deployed into a ­Kubernetes cluster are given DNS records and can thus be reached by each other via common network protocols.

For direct communication between services within our cluster, we use gRPC. Deployments can be easily replicated with Kubernetes, so multiple containers of one application can run in parallel. Requests are then distributed to the replicas via a load balancer. Crashed containers are also automatically restarted by Kubernetes. During the restart, requests are then ­redirected to containers that are still running. This automatically makes the system very stable and fail-safe. In addition, ­Kubernetes supports namespaces. A ­single cluster can thus contain multiple development and test environments.

But how does a request to the Iron ­Harvest online services work? For example, if a mission is completed while the game is connected to the online services, many different things should happen. It must be checked whether Achievements have been reached, experience points must be distributed and, if necessary, levelups with rewards must be credited. All these tasks are performed by different services. First, the game client makes a simple http REST call to the online services endpoint. A gateway service receives these requests, deserializes the JSON object and translates it into gRPC. Then, ­depending on what call was made, the appropriate target service is contacted via gRPC. In the case of a game result, it is the ­GameResult service. This then sends the game result through the cluster via a message bus. For broadcasting such messages, we use NATS. Several standalone services independently subscribe to NATS for game result notifications. If a service has multiple replicas, NATS handles load balancing. A service can respond to messages. In our example, the Achievements Service would respond after receiving and calculating a game result with a list of all newly unlocked achievements. The GameResult service, which waits for a period for responses, then finally responds to the gateway’s gRPC request. The gateway translates the response into JSON and completes the client’s http REST call.

The advantage of this approach is that it is very easy to add services that react when a game result is submitted. Also, an error in one of the services cannot lead to a complete failure of the game evaluation. The disadvantage is that a process that would otherwise have been tracked linearly in a classical architecture suddenly has many different responsibilities. Each service writes its own logs and has its own statistics.

So it’s important to start thinking early about how to keep track of things as complexity increases. All services in Iron Harvest log in JSON format. This has the advantage that other parameters can be attached to a log in addition to the classic log message. For example, for game ­results this is the ID of the game session and an anonymous user ID.

The logs of all services are collected by a Graylog server, evaluated, and displayed in dashboards. In Graylog we can then see all error logs at a glance and group the logs of all services based on the IDs.

Besides logs, it is also very important to collect statistics. How many players are currently playing multiplayer, how long does it take to calculate a score or how long do players wait for a match in ­matchmaking? Corresponding metrics were implemented by us in our services and collected by Prometheus. The data gathered in this way is then presented to a Grafana dashboard. Alarms can be defined in Grafana if a ­certain value is exceeded or not reached for a certain time. Such an alarm can result in then chat messages or similar and ruin the weekend for developers.

Matchmaking

We also opted for a Google solution for matchmaking: Open Match. This is an open-source framework that is at home in Kubernetes. It allows the user to have total control over matchmaking.

There are three services in Iron Harvest for using Open Match:

• The frontend service is addressed by the client when a player starts matchmaking. The service obtains all the information required for matchmaking and enters it in Open Match in the form of a ticket populated with metadata.
• The backend service runs continuously and asks Open Match for found games. If there are matches, the service reserves a game server and forwards IP and port to the players.
• The matchmaking functions are the core of our matchmaking. They are called by Open Match on a regular ­basis. The service queries all open ­tickets, including the metadata, and suggests possible matches to Open Match. A score is passed on how optimal this match would be. Open Match sorts the results, removes matches where tickets are duplicated and passes valid matches to the backend service.

Gameserver

Iron Harvest also uses Kubernetes to host its dedicated game servers. With Agones, Google has developed a system that is designed to manage and scale fleets of game servers. For a server to be controlled by Agones, it must implement the Agones SDK and be placed in a container. When an instance is started, it must tell the SDK when the startup process is complete, and the server is ready for use. Depending on the configuration, Agones keeps a pool of ready game servers on hand. The online services only need to ­request a server from Agones and, if ­there is still capacity available, get the IP and ports of a server. The game server is then informed by the Agones API that it has been requested and the game can begin.

The big advantage of this system is that you do not have to worry about managing servers. Agones takes care of IPs and dynamically maps the ports of the game server, so that many servers are accessible on one machine. Depending on how many multiplayer games are being played in parallel, machines can be automatically started up or shut down.

When a new version of a game server is to be distributed, free game servers are shut down and new servers with the new ­version are started up. Servers that are in use remain untouched. This makes it possible to roll out new versions during operation without the end user noticing anything.

Conclusion

The decision to use microservices for Iron Harvest has paid off for us. During ­development as well as in live operation, the servers have never let us down. ­Updates could always be installed without downtime. Even when the number of players exceeded our expectations by far during a “Free Weekend”, we were able to react very quickly to the situation. More players simply meant more hardware, which could be booted up without any problems. The load was distributed automatically, and it was possible to play without any problems.