When starting with the next version in a video game series, Halo 4, the original game engine and the available set of services couldn’t meet the requirements and a new solution was created based on the Actor model implemented by the Orleans framework. Caitie McCaffrey told in a presentation at the QCon London conference talking about the work designing and building the services supporting the new game.
One architectural challenge McCaffrey, a distributed systems engineer currently working at Twitter, earlier at Microsoft Game Studios, mentions was designing for the extensive load a game may generate. On launch day Halo 4 had 1 million unique players online and within one week 4 million with over 30 million hours of combined gameplay. In preparation for handling a load of this size the project went into using the cloud, more specifically Azure since it’s a Microsoft game. Other challenges included allowing for online playing at all times requring the servers to be always available with both low latency and high concurrency.
Looking for a solution to the requirements the project came across the actor model. Fundamentally actors are a basis for reasoning about concurrency, the core concept being an actor, a primitive for concurrent computation only communicating with one another using asynchronous message passing. When an actor receives a message it can send messages to other actors, create a new actor or modify its internal state. Keeping all data for an actor in one location effectively means McCaffrey and the team are creating stateful services allowing them to build a stateful middle tier with the performance benefits of a cache but with no concurrency issues.
Searching inside Microsoft the project found Orleans framework, a runtime and programming model for building distributed systems based on the actor model. An important aspect differentiating Orleans from other actor model frameworks is the concept of a virtual actor, an actor that can never be created or destroyed, logically it always exists. Some key ideas this virtual actor brings is location transparency, a message sent to an actor automatically gets routed to the actor irrespective of its location, and automatic scale out without redeploy.
With the new architecture McCaffrey claims that they are able to sustainably run above 90% CPU utilization on their servers with nearly linear scalability, simplifying adapting to load changes especially valuable during dramatic load increases, e.g. when a game is launched.
McCaffrey believes they would not have been successful without using Orleans, due to how easy it was to find developers when using a tool like Orleans helping them write code running in a distributed world, and in that increasing the productivity as the developers could focus on making the experience in the game really great.
In an earlier blog post McCaffrey looked into using Orleans for creating RESTful Services.
The different approaches in implementing the actor model was earlier discussed and reported on InfoQ.