At QCon London 2016 Peter Alvaro and Kolton Andrus shared lessons learned from a fruitful collaboration between academia and industry, which ultimately resulted in the creation of a novel method for automating failure injection testing at Netflix. Core learnings included: work backwards from what you know; meet in the middle; and adapt the theory to the reality.
Netflix has shed light on how the company uses the latest version of their Keystone Data Pipeline, a petabyte-scale real-time event stream processing system for business and product analytics. This news summarizes the three major versions of the pipeline, now used by almost every application at Netflix.
At the microXchg 2016 conference, held in Berlin, Rick Buskens presented “Microservice Deployment Pipelines with Spinnaker”, which discussed the collaboration between Netflix and Google on the Netflix-conceived Spinnaker continuous delivery platform. Spinnaker can be used to create build pipelines for safe and predictable deployment of microservice applications across multiple cloud providers.
Last year, Netflix Cloud Database Engineering (CDE) team introduced Dynomite. Dynomite is a proxy layer, aiming to turn any non-distributed database into a sharded, multi-region replication aware distributed database system. Now Netflix released a benchmark using Dynomite with Redis in AWS infrastructure.
Rising from the ashes of GigaOm the tribal gathering of cloud elders that is Structure has returned, and got off to a strong start with Battery Ventures' Adrian Cockcroft presenting on the State of the Cloud and Container Ecosystems. Cockcroft paid particular attention to the impact of containers, which wasn’t even a major discussion topic at the last Structure conference in 2013.
InfoQ had the opportunity to interview Daniel Jacobson about ephemeral APIs, their link to experience-based APIs and when to consider them. He also explains why generic resource-based API architectures can run into problems at scale and why he doesn’t use an API descriptor language. Finally, he describes the various tools they built to deliver those APIs including Falcor, Scryer or Nicobar.
Today, Pivotal announced an update to Pivotal Cloud Foundry (PCF), the commercial version of a popular opens-source platform for building, deploying, and running cloud-native applications. This 1.6 release gives developers native access to a subset of Spring Cloud’s Netflix OSS services, built-in support for .NET applications, beta support for Docker images, and integrated ALM tools.
Based on their experience with arbitrarily shutting down servers or simulating the shutdown of an entire data center in production, Netflix has proposed a number of principles of chaos engineering.
At QCon New York 2015, Kolton Andrus discussed Netflix’s Failure Injection Testing (FIT) platform, which allows the injection and monitoring of arbitrary failure scenarios to a targeted group of customers using the Netflix production web services. FIT allows Netflix to maintain an ‘antifragile’ programming culture, which results in the creation of systems that are resilient to failure.
The Netflix team has released FIDO -- an open source system for automatically analysing security events. Not to be confused with FIDO Alliance, Netflix's platform stands for Fully Integrated Defense Operation, the platform's Github describes FIDO as "an orchestration layer used to automate the incident response process by evaluating, assessing and responding to malware."
Recently Adrian Cockcroft gave an interview to ActiveState's John Wetherill about microservices. In it he talks about how polyglot fits into microservices and the impact on him when he head that companies such as Target and Macy's, as well as Homeland Security had adopted that architectural approach.
Netflix has open sourced Atlas, part of their next-generation monitoring platform they have been working on since early 2012. The company developed Atlas to store time series data in order to provide near real-time operational insight to teams.
Netflix has released Prana, an open-source "sidecar" application the company developed to allow heterogeneous microservice applications to use the NetflixOSS JVM-based platform support libraries.