BT

Dropbox Develops Magic Pocket, Moves Away From AWS

by Kent Weare on Mar 26, 2016 |

Recently, InfoQ published an article on Netflix completing their migration to the cloud after more than 7 years.  This article provides some contrast from that article and focuses on an organization that is moving away from the public cloud to pursue their own internal solution for delivering cloud–based file services at scale.

On March 14, 2016 Dropbox discussed their migration away from Amazon Web Services (AWS) on their blog.  The blog post recounts all of the activities that have allowed them to achieve their goal of serving 90% of all user data exclusively from their own data centers. The project, named “Magic Pocket”, called for in-house developed software and hardware to be built to support their customer needs.

Dropbox has seen 12x growth since 2012. During this timeframe, the amount of storage required for customer data has increased from 40 petabytes to 500 petabytes in order to serve their 500 million users.

Historically, Dropbox has used their own internal hardware and software to manage metadata about the files that they are storing for customers.  The actual data resided in AWS S3 storage containers. Dropbox developed software would then merge these two data sources together in a single application.

Dropbox credits Amazon for their ability to help them scale and grow as fast as they have, but Dropbox felt they could further optimize and run their own data centers more cost effectively. Akhil Gupta, ‎VP of engineering, infrastructure at Dropbox, further explains the rationale behind bringing their cloud-based file service in-house: “There were a couple reasons behind this decision. First, one of our key product differentiators is performance. Bringing storage in-house allows us to customize the entire stack end-to-end and improve performance for our particular use case. Second, as one of the world’s leading providers of cloud services, our use case for block storage is unique. We can leverage our scale and particular use case to customize both the hardware and software, resulting in better unit economics.”

Dropbox felt they were in uncharted territory with only “a handful of exabyte-scale storage systems in the world.” They felt there was nothing in the open source community that was reliable or could provide the necessary scale so they built their storage system from scratch. Some of the requirements they established for themselves included data encryption at rest, annual data durability of 99.9999999999%, and availability of over 99.99%.

A small team at Dropbox started coding Magic Pocket in the summer of 2013. The team started with a small prototype that would eventually need to become one of the largest storage systems worldwide. Just over a year later, in September 2014, the team embarked on a “Dark Launch” that allowed the organization to start mirroring data from 2 regional locations. On February 27, 2015 Dropbox went ahead with their “Launch Date” that saw them exclusively using their infrastructure to serve file metadata and contents.  Over the following 6 months they continued to scale their load to 500 petabytes. Internally, Dropbox had a deadline of moving 90% their entire workload to their own custom solution.  The team was able to hit this deadline early and achieved this milestone on October 7, 2015.

For the remaining 10% of their workloads, Dropbox will continue to partner with Amazon where it makes sense.  One particular use case is in Europe, where workloads are not as predictable or where privacy laws may not make hosting the workload internally feasible.  Gupta highlights one of these particular use cases: “We’ll continue to invest in our own infrastructure as well as partner with Amazon where it makes sense for our users, particularly globally. Later this year, we’ll expand our relationship with AWS to store data in Germany for European business customers that request it. Protecting and preserving the data our users entrust us with is our top priority at all times.”

Dropbox is not the only Silicon Valley company to move workloads from the cloud, in-house.  Back in 2012, Zynga spent over 100 million dollars on deploying their own infrastructure and subsequently bringing public cloud workloads in-house. The move was highly publicized and at the time served as a case study for moving workloads away from the cloud.   In May 2015, Zynga changed their course only to start putting workloads back in AWS. On an investor’s call, CEO Mark Pincus indicated that “There’s a lot of places that are not strategic for us to have scale and we think not appropriate, like running our own data centers, we’re going to let Amazon do that.”

Whether Dropbox will become the next Zynga remains to be seen.  InfoQ will continue to monitor these developments and report on any updates.

Rate this Article

Relevance
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss
General Feedback
Bugs
Advertising
Editorial
Marketing
InfoQ.com and all content copyright © 2006-2016 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT

We notice you’re using an ad blocker

We understand why you use ad blockers. However to keep InfoQ free we need your support. InfoQ will not provide your data to third parties without individual opt-in consent. We only work with advertisers relevant to our readers. Please consider whitelisting us.