BT

How Netflix Deploys Code

by Zef Hemel on Jun 13, 2013 |

Netflix, the popular movie streaming site, deploys a hundred times per day, without the use of Chef or Puppet, without a quality assurance department and without release engineers. To do this, Netflix built an advanced in-house PaaS (Platform as a Service) that allows each team to deploy their own part of the infrastructure whenever they want, however many times they require. During QCon New York 2013Jeremy Edberg gave a talk about the infrastructure Netflix built to support this rapid pace of iteration on top of Amazon's AWS.

Netflix uses a service-oriented architecture to implement their API, which handles most of the site's requests (2 billion requests per day). Behind the scenes, the API is separated into many services, where each service is managed by a team, allowing teams to work relatively autonomously and decide themselves when and how often they want to deploy new software.

Netflix is heavily invested in DevOps. Developers build, deploy and operate their own server clusters and are accountable when things go wrong. In case of failure, a session is organized where the root cause of the issue is investigated, and ways are discussed to prevent similar issues in the future -- similar to the five whys.

Deployment at Netflix is completely automated. When a service needs to be deployed, the developer first pushes the code to a source code repository. The code push is picked up by Jenkins, which subsequently performs a build producing an application package. Then, a fresh VM image (AMI) is produced based on a base image (containing a Linux distribution) and software that all Netflix servers run, including a JVM and Tomcat, possibly further customized by the team. On top of this base install, the application package is installed. From this, an AMI is produced and registered with the system.

To deploy the VM images to its infrastructure, Netflix built Asgard. Via the Asgard web interface, VM images can be instantiated to create new EC2 clusters. Every cluster consists of at least 3 EC2 instances for redundancy, spread over multiple availability zones. When deploying a new version, the cluster running the previous version is kept running while the new version is instantiated. When the new version is booted and has registered itself with the Netflix services registry called Eureka, the load balancer flips a switch directing all traffic to the new cluster. The new cluster is monitored carefully and kept running overnight. If everything runs OK, the old cluster is destroyed. If something goes wrong, the load balancer is switched back to the old cluster.

Failure happens continuously in the Netflix infrastructure. Software needs to be able to deal with failing hardware, failing network connectivity and many other types of failure. Even if failure doesn't occur naturally, it is induced forcefully using The Simian Army. The Simian Army consists of a number of (software) "monkeys" that randomly introduce failure. For instance, the Chaos Monkey randomly brings servers down and the Latency Monkey randomly introduces latency in the network. Ensuring that failure happens constantly makes it impossible for the team to ignore the problem and creates a culture that has failure resilience as a top priority.

Many parts of the Netflix infrastructure are open source already and available on Github. It is Netflix' goal to eventually release all of its infrastructure for other companies to benefit from.

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

how about the data tier? by Chen Island

Are the old version and new version operating on the same "DB"?

If the new version requires a different DB schema, how dose Neflix deploy the new code and the new schema with the old data?

Re: how about the data tier? by Suresh G

this might be an issue when using relational db. Can't this easily managed in the nosql world?

What about DB? by First name Last name

How do they copy the production database to the new envoriment?

Thanks!

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

3 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT