Sure. My name is Nick O'Leary and I work in the Emerging Technologies Team at IBM. So we are a small team who work in the space of new innovative technologies, looking, one to two years in the future. We also do some fundamental research work, a huge variety of different types of things really.
Chris: Nick you just did a presentation at QCon on Node-RED and people are going to be able to see the video of that in its entirety on the InfoQ site in due course, but tell us a little bit about Node-RED and how you came to be creating that.
Sure. So Node-RED is a project we started about a year ago now to create a tool to make it easy to – we say – wire the Internet things. And what we mean by that is to give the development environment that isn't about writing code to talk to a lot of different systems. It isn't about trying to work out all the fine grain details of “how do I talk to a serial device” or “how do I do an OAuth flow to authenticate into Twitter”, but to enable you to just – in a graphical environment – just drag on an element and say “this is a serial port”, and drag on one that says “this is a Twitter stream”, and be able to draw a link between the two in the browser, to start wiring together these different streams of events you can get.
It started out as an exercise in learning new technologies. Because being an emerging technologies team one of the things we need to do is try to stay current, try to stay ahead of where things are going. So, as I said, about a year ago, I had some time away from client paid for client work and I wanted to start learning some new technologies. So node.js was top of my list to start playing with, and as I started playing with it, it soon became apparent that it was well suited to solving this problem of how do you create a nice modular system to talk to different devices, different streams, and it just became an obvious choice really. I think, given its event-driven nature of Node, of javascript, it just seemed an obvious choice.
In many ways I was already looking in node.js and it just became the obvious choice to solve this problem. I may have to say Node-RED is about the third or forth time over the last five or six years I have set out to create a visual environment for wiring different things together. Each of these previous situations have never got beyond of being able – in a web browser - to drag some boxes on the screen, draw a line between them because that’s just been so hard work in previous versions, previous browser capabilities. And this go around, using the D3 visualization toolkit in the browser. I very quickly created the visual tooling side of it.
It came together quite nicely. And working with node.js on something else, you know, playing with it, a colleague said well, all well and good that you’ve got drawing things in the browser, now make it actually do something when you hit the deploy button. So I went away and just wrote the node.js runtime side of it and it all started coming together and has just grown from that point.
Yes. So when you look at Node-RED you got the main workspace where you're doing the drawing. That is all D3, it's a Twitter bootstrap aap, just because that makes it easy just to do the basics, or skeleton of the page. It's jQuery for doing some of the UI's, dialogs and such things. I think one of the more interesting bits we use is the Eclipse Orion javascript editor. So one of the nodes you can drag into Node-RED is a function node. But when you edit it, it gives you a rich text editor for writing javascript. That element, it started out as just a plain text area but working with colleagues who work on Eclipse Orion they pointed us that their browser-based, very rich text editor for programming languages. So it took five minutes to embed their editor into Node-RED. It suddenly gave us syntax highlighting, line numbering, auto complete, just little things that you take for granted within an IDE, you've now got in a browser environment. Again, it's just little things, that was really cheap to do, cheap in terms of literally five minutes work and five lines of code if that, to change a text area into this rich, language aware text editor. So, definitely an interesting one.
Chris: So, it seems like Node-RED itself is almost a Lego-esque paradigm for creating these different flows between various types of input and output. And I think, as you've just described, you were able to build a number of Lego-esque components that now exist mostly out in open-source land. Tell us a little about the flows-library and how people now are able to share semi completed models
Sure. So one of the things within Node-RED from the race start was to try make it really easy to do social development of your flows. That your flows are just representatives JSON text. When you hit that deploy button it's squirting JSON back to the runtime. In the editor you can highlight any node you want, export the JSON to your clipboard so you can send over email to a colleague or you can share it, or you can just save in the archive off locally or whatever you want to do with it. And that was really powerful. I found with a colleague when we're working on real client projects using Node-RED, the fact that we could quickly just bounce, the JSON flows between ourselves over the instant messaging at work. Really made a powerful tool for that social development. And we wanted to make that more accessible and take advantage of that from the open-source community aspects in Node-RED. So over the Christmas break I put together the flows libraries, so this is part of the Node-RED website that anyone can log in using a GitHub account and they can paste in a flow that they've created and give a description and add it to the library.
And you can tag it with keywords to help people find your flow. And it means people can start sharing things they've done. We did a soft launch, it's only recently we've gone public, so there's a growing number of things appearing in there but it's fascinating to see how other people are using Node-RED, that we have our use cases that we use it for a lot, and it tends to be run on a Raspberry Pi, talking to sensors or whatever it might be, but you know, people got it running. There's one guy who had set and created a flow and shared on the flow library that he will get a text message if there's an overnight frost, to wake him up fifteen minutes earlier, so he can defrost the car. Now you see, that's a flow specific to him, because it's based on where he lives, his phone number, that sort of thing. But just by sharing it with placeholders for those bits anyone can take advantage. It's these recipes for people to learn from and build on, and collaborate on.
Sure. So one of the early projects we used Node-RED with was a manufacturer who wanted to instrument a production line with a whole bunch of sensors for a proof of concept, for some problem they were trying to solve. So we went in with and we ended up strapping three Raspberry Pis to this factory line with USB webcams so we could take photos of things on the line. And then each Pi has an Arduino attached, with whole bunch of sensors: temperature, humidity, pressures, whole range of things. And we needed some way to orchestrate, when we get signal that the production line has shifted one product through, to trigger reading all the sensor data and all the different actions we had to take. So we ran Node-RED on each of those Raspberry Pi's using our UDP, input nodes and output nodes. So one Raspberry Pi has a signal to a GPIO pin, from the machine itself.
When Node-RED detects that signal it triggers a flow that does a UDP broadcast, so the other Raspberry Pis get the signal to say yes, start capturing your data. All the Pis then talk over serial to their Arduinos to trigger data collection, they trigger the webcam. And when all that data comes back to the flows send all that data back up to a central, more typically enterprisey central system, where processing of analytics, that type of stuff can happen. And that was really powerful for us the fact we could very quickly orchestrate all these different sensors, because we went in not knowing what sensors we are going to find, what their requirement were. So having this platform that we could very quickly just plug new things in, change what was there, was really powerful.
One great advantage they brought us was when they phone up to say they have had to change one of the sensors because of a hardware fault and now the reading we would get, the voltage reading we got, maps to a different pressure range. So they are asking, you know, big system, how long is it going to take, how soon do you think you could update system? Whilst we're on the phone, I logged in to the network, I pointed my web browser at the appropriate Raspberry Pi, just opened-up the function node in Node-RED, and in my browser typed in the new figures he'd given me and hit deploy, and before we had hung-up the phone, the system was updated. When you’re working creating proof of concepts, trying to be very agile and react to that sort of change, having a tool that made it so easy to do that was invaluable.
In fact, for the same people, they had a builder in for a couple of weeks renovating an office. And they kept turning off the server we were using. And we were the only ones using that server accessing it externally so no one internally would notice. Now of course we could have installed server monitoring software to alert us, but it was only, the builders only were there two weeks. We just needed a quick solution for a very short period of time. So in the instance we ran in the lab, I just set up a flow that poled, it did an HTTP GET of the server every now and then, and if it ever got back an error code, the flow sent off an email to five people, so you know, a couple of the clients and a couple of us, just saying the server is down, can someone turn it back on, please? So it’s that very bespoke thing that was only needed for those two weeks but solved a real need, it was really useful. In terms of other things we've seen people do with it, I mean, it’s a great variety, we're always amazed when we see what other people see in the tool and how they can turn it to their own needs. I mentioned the guy who set up the checking the weather forecast so he would get a tweet.
There's things around, setting up alerts onto home energy monitoring, knowing when your energy has gone over a certain usage for too long, have you left lights on, you shouldn't, that sort of thing. I think where it gets quite interesting is where it bridges the physical-digital divide, when you are talking about running on the device with physical things attached to it and getting them to be sort of active online devices, I mean true Internet of things devices. A nice example was a colleague that got – I think – a Freetronics LED cube, one of these five by five cube of LED's, you can make go all sort of colors. He very quickly wired that up to be a brilliant visualization of energy use of a whole community where each LED was a different house in this community. In fact he could create this physical visualization of that data just with wiring together a few boxes with a bit of logic to map different houses to a different LED on this device. It’s incredible to see.
5. You recently released Node-RED version 0.6, what gets you to 1.0?
I don't know. I don't know if we would ever get to 1.0. It's a living breathing tool at the moment. We have a list of things we want to do with it internally, new features, new capabilities but then equally as an open-source project with quite a vibrant community, we do get that feedback of what other ideas people are bringing into it and what they would like to say it do. So I think...there's part of me that thinks “I don't want to get to 1.0 because that almost implies that we’re finished”, and as a tool has a quite vibrant development behind it, you know, we're actually working on it. I like this idea that we want be adding new features as we go along but bringing value not just for the sake of it and I think trying to stick an arbitrary flag in it and saying “right this is 1.0” doesn't quite fit. But who knows? It is early days. I mean we open-sourced it in September last year. That's six months now and in the early days, for the first three or four months we were doing a release regularly every two to three weeks. It’s slowed down a bit now, I'm just, you know, time issues, other demands on the time at the moment but I think there's a lot still to do.
Chris's full question: We were just talking to Adrian Cockroft who was complaining about the weather station he got for Christmas having a closed interface after previous versions that had open interfaces. Is that the trend going in the wrong direction? Are you expecting to see more open-source hardware and for devices to be more hackable and therefore more easy to integrate into environments like Node-RED as we go forward?
I’d certainly hope so, leaving aside the open hardware side of it, I think even if it is an entirely proprietary hardware device, having an API so you can access the capability of the device, is almost a no-brainer. Particularly with IoT, because...there is this idea that the devices you have, consumer devices you have in IoT are just the service avatar of some online service. And if that online service goes away, you're left with just this inert blob of plastic and metal that doesn't do anything anymore. So I certainly think having devices that have open APIs at the very least, gives them a life beyond the life of whoever produced it. And I think with so many people producing IoT devices it's going to be interesting to see how the trend changes over time, just how open these devices are to be reused and re-appropriated to other uses. And certainly a tool like Node-RED, trying to act like an integrator between the different APIs, if a device has an open API, it's all the more easy for it to become part of the palate of capabilities in Node-RED. You look at some devices like the – I did mention in my talk – the WeMo sockets which didn't have a formal API for interacting with but people have found ways in. So there are node modules for talking to the WeMo sockets. Recent, announcements from them suggest they may be closing down those APIs – not even APIs – there's backdoors people found and with a promise of an open API to follow. But we'll wait and see what happens. I think open APIs is, as I said, it almost feels a no-brainer. The value in a device in the service you offer on top of it and the API shouldn't be this black box proprietary system.
7. [...] Do we think we are going to see an advent of things that talk MQTT?
Chris's full question: So in addition to APIs, because it's consideration about protocols in his evening keynote yesterday Tim Bray gave something of a state of the nation on software and he talked about HTTP winning. But a lot of what you've done with Node-RED is based on MQTT. So do we think we are going to see an advent of things that talk MQTT?
I’d certainly hope so, leaving aside the open hardware side of it, I think even if it is an entirely proprietary hardware device, having an API so you can access the capability of the device, is almost a no-brainer. Particularly with IoT, because...there is this idea that the devices you have, consumer devices you have in IoT are just the service avatar of some online service. And if that online service goes away, you're left with just this inert blob of plastic and metal that doesn't do anything anymore. So I certainly think having devices that have open APIs at the very least, gives them a life beyond the life of whoever produced it. And I think with so many people producing IoT devices it's going to be interesting to see how the trend changes over time, just how open these devices are to be reused and re-appropriated to other uses.
And certainly a tool like Node-RED, trying to act like an integrator between the different APIs, if a device has an open API, it's all the more easy for it to become part of the palate of capabilities in Node-RED. You look at some devices like the – I did mention in my talk – the WeMo sockets which didn't have a formal API for interacting with but people have found ways in. So there are node modules for talking to the WeMo sockets. Recent, announcements from them suggest they may be closing down those APIs – not even APIs – there's backdoors people found and with a promise of an open API to follow. But we'll wait and see what happens. I think open APIs is, as I said, it almost feels a no-brainer. The value in a device in the service you offer on top of it and the API shouldn't be this black box proprietary system.
I think...well, there are a number of close source binary blobs on there, less today than there was a week ago, because I think Broadcom have just announced that they are open-sourcing a bit of the graphics driver. I think when you compare the two, I mean there's not much comparison to be made. When you look at an Arduino, I know I can look at the schematics. And as a hobbyist, solder up my own clone of an Arduino to do stuff. The Raspberry Pi, there's a lot of...yes, my soldering isn't that good. In order to go off and… you simply couldn’t they're able to take advantage of relationships they've got to get at scale, things such an individual couldn't do. So that appeals to me, as someone who likes to build things from scratch. But that's not to say and the Raspberry Pi is an absolutely great platform and the Beaglebone, mbed, you know, there's a great range of them and I don't get too hung up on being fully open hardware platforms. I think as long as they are open in the right way – if that make sense? I don't personally feel the need to see a schematic as long as I can get on and do what I want, not have to be constrained by the environment. To me that's a good thing.
Chris: Yes. I sometimes myself think about the toaster projects, so you may have seen somebody wrote a book about making a toaster from scratch and mined his own iron ore and everything else and I think it's too easy for open hardware projects to turn into toaster projects. If you insist on everything being open, because you inevitably end up using somebody's intellectual property somewhere. And I think some of the differences between Arduino and Pi can just be about the how agricultural or not the hardware is.
Yes.
Chris: So Nick, thanks very much for spending some time with us, that's been absolutely great and I hope you enjoy the rest of the conference?
Thanks Chris