Netflix's Reactive Programming Model via Rx

1. Hi this is Harry Brumleve, I’m at QCon San Francisco 2013, I’m with Jafar Husein from Netflix. Jafar could you tell us a little bit about yourself?

Yes, I’m a tech lead at Netflix, I work across of all the UI Teams and I’ve about 12 years in the industry, I worked previously at the Microsoft and GE and I’ve got about 4-5 years under my belt building Functional Reactive Systems.

2. That is the subject of your talk here, could you tell us a little bit about that?

Yes, here I’m talking about Functional Reactive Programming in JavaScript, I’ve been talking a little bit about contrasting the different approaches to Functional Reactive Programming in JavaScript and specifically I’ve been talking about the way we approach it at Netflix.

3. And so Netflix has a large investment in Reactive Systems, why are they important?

Well they are important in Netflix because we scale and we really scale, at any time we are responsible about a third of US Broadband traffic, and so it’s really important for our systems to be responsive. Now that is important on the backend, on the frontend we build UIs, all UI’s are reactive and so we want to be able to turn around AB Tests as quickly as possible and so it’s really important, we embrace a library that allows us to do that quickly despite the difficulty involved in building Reactive Systems in JavaScript today.

4. And so Netflix uses RX’s one of those tools to achieve Reactive Systems, could you tell us a little bit about that?

I was working at Microsoft about 4 years ago and I have the good fortune of running into Erik Mayer who is the father of the Reactive Extensions. At that time I was sort of struggling with a tough problem which was, we are doing Asynchronous Unit Testing for Silverlight which is UI Framework and one of the challenges involve with testing UI Frameworks is that there is asynchronous rendering ware so you can layout widgets but you have to wait for the Asynchronous process to complete before you can really ensure that everything that work is correctly. So as a result to solve that problem we ended up putting a bunch of timing based approaches in to sort of make sure that we wait a sort amount of time and hopefully the rendering process to be complete and we check it. Now of course that introduce a bunch of false negatives and it was a real headache, and so what I really wanted to do was come up with something more reliable. Now we knew we could listen for events to figure out and know when that rendering layer had finished and then go in and test things.

The only problem was listening to all those events and registering call backs, turned our code in an absolute mess and so I was really searching for a better option and as I said I had a good fortune of running into Erik because at that time he was working on something absolutely revolutionary, a very revolutionary idea which was that events and asynchronous operations in general could be modeled this collections and I was absolutely floored because I was able to use sort of the operations that I already understood like Map, Filter and Reduce and that I learn from say things like SQL and I was able to apply them to event streams to build really, really clean declarative code. And so I was absolutely amazed but what it can do and I thought this will be great to build an entire system on. So fast forward a few years I’m now at Netflix and we’ve essentially rewritten most of our reactive code on many of our devices to use RX. We are also using on the server so we are actually doing an end to end reactive system, we are using across three different platforms and it’s absolutely transformed the way we think about asynchronous programming and reactive programming at Netflix and the way we do it every day.

5. What is the model that RX uses in order to transport that data back and forth?

RX asks a very interesting question which is: “What is the difference between a SQL query and a mouse drag event”. Now really if you think about both these things are Collections, the only difference is, and in both cases you’ve got a data producer giving a data consumer a piece of data a little bit by bit, right? The main difference is that in a SQL Query I’m pulling data as a producer from a consumer but in an event I’m pushing data, the producer is pushing data to the consumer. So it’s really the same thing the data just travelling in a different direction and there are all sorts of interesting operations that we know how to do in SQL like, select where a Map and Filter if you will and it turns up that all of the operations can also be performed on events, so you can write very declarative code to control Concurrency and coordinate Asynchronous operations and essentially it has the same meaning whether you are working against an in-memory collection or an event, and that was the revolutionary idea that really transform the way that I thought about Asynchronous Programming and that is what we do, now that transforms a code that we write every single day.

6. And that sound like a pretty significant paradigm shift for a typical software developer to undertake, in order to use RX do I have to become a polyglot programmer?

No, all languages that you really know are orthogonal. We use it across 4 different languages and several platforms but it’s definitely a paradigm shift to think about asynchronous processes as Collections, but really I liken a process to sort of recovering from brain damage. I sort of been dealing with call backs for so many years, it was hard to make the transition but I think that if you are sort of introduce the idea of events as Collections from the very beginning you would down the process very easy. So in many ways is unlearning those bad habits that sort of leads you down a better path and that does take some time to get over. Once you get over there you can do absolutely amazing stuff, ninja stuff, now we’ve got a fantastic group of developers on the UI Team who can all write circles around me in terms of Asynchronous Programming, now that they’ve learned this stuff and they are using in everyday.

7. If I wanted to establish a Reactive System’s practice on my team, how would I begin?

There is really two key pieces: training and support. So one of the key pieces from us was making sure that we were able to introduce our developers to Reactive Programming in a really gradual way. It’s difficult to give developers who are on tight deadlines, technology and just ask them to build the system. You’ve got to give them a great training program so they get up and speed quickly and you’ve also be there to support them, if they run into roadblocks and using the technology they are not sure about the right way to approach from a design perspective. So I’ve already have several years of Reactive Programming so I was able to be that point person until other members of the team sort of boots strapped up. Now there is a huge number of people at Netflix with competency both on the server side and on the client side with RX, and so now that scales much better. As far as training is concern, we actually build a set of online interactive exercises that are client developers and server developers could fill up in JavaScript.

Now what we actually do is we take them gradually through the process of using imperative constructs like Loops and If Statements to using Functional Programming construct over Arrays, so Map, Filter, Reduce, Merge, Zip, five functions, and after a while they learn how to build complex expressions and queries over arrays using those 5 functions and then we play a little trick on them, we flip the switch and we hand them this observable type that is really sort of like an event, an object is like an event, and all the same code that they’ve written that works over arrays, works over this observable and now it’s sort of comes alive instead of just producing all the data, all the sudden you start to see data trickle in, instead of writing a query over a bunch of stock legacy historical stock information, they are actually writing a query over realtime stock data and they didn’t even know it until they press the run button. That process of sort of taking them gradually from imperative to functional reactive has been really successful for us.

8. So it does sound like there is a significant amount of investment in both learning and training in order to achieve Reactive System practice. How do I convince my boss that is a good idea?

Well I want to be clear, let’s face it, if I take an observable and just hand it to you and I say: “Well you can register an event handler with this and then you can call this method to unregister an event handler”. It’s no more unless complex than an event so the learning curve here, if done correctly, can be really ramped up slowly. The idea is to introduce these observable data sources into your system and at first developers can approach it just like an event, they are no were soft that they went before with DOM events or XHR callbacks. But then over time as these observable data sources become more and more prevalent in the system, that gives them time to learn how to compose them using these powerful operators, Map, Filter, Reduce, Merge and Zip and so what we’ve done is we’ve created this environment where there is now a tremendous incentive for them to learn because they see what their peer developers are doing and they see how they can write very, very low bug count code with just a few lines of code they do very, very powerful things, and so a culture starts to develop.

So I don’t want to give the impression that this is a high learning curve, if done correctly and introduce gradually this process can be very, very smooth, and that is exactly what we’ve seen at Netflix. We are taking UA developers who’ve never done any Functional Reactive Programming before and we are having them write really powerful systems and integrate with really reactive systems very quickly within 2 or 3 weeks. Now that doesn’t mean they’ve learned everything about how to do Reactive Programming but then is that when the training programs take over and mentoring from other folks there who have Reactive Programming experience. Over time they start to really shine and they start to really want to do more and more Reactive Programming. I’m seeing tremendous enthusiasm from people who’ve been on the team for 2 or 3 months. At first they might have been a little bit iffy on the technology, now they are coming to me and asking me how they can build our search system that way, how they can do data binding with RX. So in terms of convincing your boss, definitely you can take the training program that Netflix has out there and you can just drop a men and start using them today. The important thing it’s to get the framework into the system so people can start using it because really what you need is you need to start building up those observable data sources, stop producing callback API’s and start producing API’s as return observables. You won’t be any worse for wear for doing that but you are building more value, more incentive for developers to suddenly really take hold for what the framework can do and take it to the extreme.

Harry: Thank a lot for your time Jafar!