Dropbox Develops Magic Pocket, Moves Away From AWS
Recently, InfoQ published an article on Netflix completing their migration to the cloud after more than 7 years. This article provides some contrast from that article and focuses on an organization that is moving away from the public cloud to pursue their own internal solution for delivering cloud–based file services at scale.
On March 14, 2016 Dropbox discussed their migration away from Amazon Web Services (AWS) on their blog. The blog post recounts all of the activities that have allowed them to achieve their goal of serving 90% of all user data exclusively from their own data centers. The project, named “Magic Pocket”, called for in-house developed software and hardware to be built to support their customer needs.
Dropbox has seen 12x growth since 2012. During this timeframe, the amount of storage required for customer data has increased from 40 petabytes to 500 petabytes in order to serve their 500 million users.
Historically, Dropbox has used their own internal hardware and software to manage metadata about the files that they are storing for customers. The actual data resided in AWS S3 storage containers. Dropbox developed software would then merge these two data sources together in a single application.
Dropbox credits Amazon for their ability to help them scale and grow as fast as they have, but Dropbox felt they could further optimize and run their own data centers more cost effectively. Akhil Gupta, VP of engineering, infrastructure at Dropbox, further explains the rationale behind bringing their cloud-based file service in-house: “There were a couple reasons behind this decision. First, one of our key product differentiators is performance. Bringing storage in-house allows us to customize the entire stack end-to-end and improve performance for our particular use case. Second, as one of the world’s leading providers of cloud services, our use case for block storage is unique. We can leverage our scale and particular use case to customize both the hardware and software, resulting in better unit economics.”
Dropbox felt they were in uncharted territory with only “a handful of exabyte-scale storage systems in the world.” They felt there was nothing in the open source community that was reliable or could provide the necessary scale so they built their storage system from scratch. Some of the requirements they established for themselves included data encryption at rest, annual data durability of 99.9999999999%, and availability of over 99.99%.
A small team at Dropbox started coding Magic Pocket in the summer of 2013. The team started with a small prototype that would eventually need to become one of the largest storage systems worldwide. Just over a year later, in September 2014, the team embarked on a “Dark Launch” that allowed the organization to start mirroring data from 2 regional locations. On February 27, 2015 Dropbox went ahead with their “Launch Date” that saw them exclusively using their infrastructure to serve file metadata and contents. Over the following 6 months they continued to scale their load to 500 petabytes. Internally, Dropbox had a deadline of moving 90% their entire workload to their own custom solution. The team was able to hit this deadline early and achieved this milestone on October 7, 2015.
For the remaining 10% of their workloads, Dropbox will continue to partner with Amazon where it makes sense. One particular use case is in Europe, where workloads are not as predictable or where privacy laws may not make hosting the workload internally feasible. Gupta highlights one of these particular use cases: “We’ll continue to invest in our own infrastructure as well as partner with Amazon where it makes sense for our users, particularly globally. Later this year, we’ll expand our relationship with AWS to store data in Germany for European business customers that request it. Protecting and preserving the data our users entrust us with is our top priority at all times.”
Dropbox is not the only Silicon Valley company to move workloads from the cloud, in-house. Back in 2012, Zynga spent over 100 million dollars on deploying their own infrastructure and subsequently bringing public cloud workloads in-house. The move was highly publicized and at the time served as a case study for moving workloads away from the cloud. In May 2015, Zynga changed their course only to start putting workloads back in AWS. On an investor’s call, CEO Mark Pincus indicated that “There’s a lot of places that are not strategic for us to have scale and we think not appropriate, like running our own data centers, we’re going to let Amazon do that.”
Whether Dropbox will become the next Zynga remains to be seen. InfoQ will continue to monitor these developments and report on any updates.