Continuous Delivery Challenges in Mobile Development
Jesper Richter-Reichhelm, Head of Engineering at Wooga, spoke at GOTO Amsterdam 2014 about some of the challenges teams face developing mobile games with a continuous delivery mindset. In particular Jesper stressed how lack of control over the software delivery process on mobile nearly crashed their business.
Jesper illustrated his point with a new release of a mobile game on the App Store late last year. The release took 4 days before being accepted by the App Store but contained a critical bug which impacted 15% of the userbase. Although it took only a few hours to fix the bug and send a new release to the App Store, the actual users only got the new version 5 days later as the period coincided with Thanksgiving holiday in the US. Thousands of bad reviews and nearly 200.000 users affected nearly killed the game and potentially the whole company, said Jesper.
Increasingly common practices that support continuous delivery strategies in the web such as canary deployments (for controlled rollouts of new features) or feature toggles are not supported by the App Store release process. To bypass those limitations, Wooga invested on supporting online configurations for their games and also recurred to some tricks like “artificial” device requirements (e.g. having a camera) in order to segment their users and be able to canary deploy some features. Still, the team depends on users to install the application updates and up to 10% of the user base remains on 18+ months old versions. Some would use the app offline when faced with forced updates to play online.
Furthermore, Wooga discontinued manual release builds from a dedicated machine for delivery to the App Store (but Jesper recommends storing dSYM file in version control for debugging crashes) as those constituted a single point of failure that increased the time to repair in case a new build needs to be sent to the App Store. Also an internal app store was set up to mimic the release process and allow internal users to test new versions of the games (in a similar fashion to Facebook).
Besides the delivery mechanism and MTTR (Mean-Time-To-Repair), Jesper also highlighted the lack of control in terms of network and devices as major challenges in mobile development at Wooga.
Particularly mobile network latency fluctuations and offline usage can lead to data loss which in turn makes it hard to keep data consistency and/or debug problems faced by users. Wooga tried to minimize this problem by adopting asynchronous communication with the mobile clients, treating all messages (even days old) equally and including ETags for solving update conflicts.