We are here at Qcon 2008, in London. I'm sitting here with Joe Armstrong, designer of Erlang. Let's talk a little bit about Erlang, which I think is your main topic. What's your current involvement with Erlang?

Today I go round and give talks about Erlang, promoting Erlang - that's one side of what I do. I've written a book on that language, again a sort of promotional thing. I think I am about 40% author and 30% programmer and 15% entrepreneur, I don't know if that's up to 100%, I haven't kept track, but I like writing and programming and solving problems. I don't like the paperwork and the business side of things, that's boring, but technical things I like.

Do you actually develop the Erlang system now? Who does that now?

That is done by a group of people at Ericsson. It is a group that maintains and supports it.

Erlang has seen quite a rise in popularity, a big rise. We always hear about the famous phone switches and I am not going to ask you about those. Who else is using Erlang? What do you know about people who use Erlang?

I know about the companies who are using it. I don't know so much about the individuals who are using it. And I don't know how many people downloaded. When we released a new site we got 10,000 - 20,000 downloads immediately, but then people package it for their Debian and for the Mac and for things like that. It goes through Mac ports and it goes through the Debian and I have no idea how many of the secondary takers are. The book sells very well, so I don't know if everybody buys the book, then goes and downloads it and tries things out. I do have a better idea of the companies that are using it.

What kind of companies? Can you say?

Yes, in Sweden there are 3 companies which are using Erlang at the moment and it's a company called Kreditor, which is in financials; there is a company called TLF, which does network management systems and there is a company Synapse, which does provisioning of mobile phones. So they are the main Erlang companies in Sweden. Each one of them employs about 30 people and they are probably market leading in each of their areas, very niched areas. This kind of thing's happening in these days, I mean Amazon using Erlang for this SimpleDB and things like that.

Is that a fact?

I believe it to be true. I haven't talked to the person who has done it and that seems to be so circumstantial evidence that it is. Amazon has a lot of services, they don't say "S3 is implemented in C++ or Java". It's just a service, you don't know what it's implemented in. It is claimed that it is written in Erlang, so I don't know if it's true or not. Certainly it has an interface that matches nicely with Erlang, so it's probably true. But then I don't know.

Erlang is good at concurrency, it was designed for it. Concurrency is obviously important now with the whole hysteria about multicore and so on. Do you know why Erlang got so popular and not other languages like ADA or OCCAM or other concurrency focused languages?

I've talked to Ralph Johnson and he said "You are not plugging the fact that Erlang is good at concurrency" I said "Yes, I am. I tell people it's good at concurrency." And he said "Yes, but it's the only language that does concurrency properly." I said "What do you mean?" He said "All the others don't do it properly." There is a sense in which if you can spawn or create millions of processes then you are doing concurrency properly. If it's limited, if you can do 10,000 or 20,000 processes, you are not doing it properly. And most languages don't do concurrency properly. It's not really baked into the language right at the bottom. What they are using there is the concurrency of the underlying operating system. Most languages, when you say "concurrency" you talk about a thin layer on top of the operating system. C++ processes or threads are just mapped onto the operating system process, which is a very expensive mechanism. In Erlang the notion of a process is part of a programming language, is not part of the operating system. There is a fundamental difference there.

Are there any theories that it's based on? Like a pi-Calculus or anything like that?

It's not based on the pi-Calculus, it grew out of some extensions to Prolog, just adding processes and message passing to Prolog. The theoretical basis would Actors model of computation, Carl Hewitt, and I don't know when that was - the 70s or so. But, if you look at concurrent languages, there are these 2 families, there is shared memory processing and there is message passing concurrency, and Erlang is a pure message passing language, which copies everything. You could think of it like e-mail or something like that. E-mail is something, which works with mailboxes. You send a message, it arrives in your mailbox and it's quite easy to structure things. If you could solve your problem by sending thousands of e-mails, then you've written an Erlang like architecture and then you do that in the language itself.

You just mentioned that Erlang grew out of Prolog, you built Erlang in Prolog, I think. What is the general development model of Erlang? How does a new feature get added to the language or the system?

I think we'll see very few changes in the language itself. We'll see changes to the libraries and things like that, because now we've got to the point where there are millions and millions of lines of legacy code, which is all tested and people don't want to muck around with that. I suspect there will be very few syntactic changes to the language. You get one chance to get it right when you make it and then when there are 10 millions of line of code written, you've blown it because you can't change it anymore - I think we are there from that point of view. Of course, the libraries can do that. Then you can make limited changes to the syntax, provided they are backwards compatible, don't break o old code. Every time you change it, you restrict your freedom even more to change it in the future. The awkward thing is there is no good mechanism for removing old stuff. Niklaus Wirth said... "People come and say to me 'Can't we add this to Erlang" and I say "Yes, it's great. What are we going to remove?", because I want to still keep it simple. If you have the goal of a simple language, when you add stuff you got to take stuff out, but people are incredibly reluctant to remove stuff, because it will break all their old code and they don't really want that - difficult!

What would be some features or some changes that you would like in Erlang? Some changes that you would like in Erlang? Let's forget the legacy code.

I would like this data structure called a HashMap, the sort of thing that you got Python and Ruby that base a lot of things on HashMaps, it's like Erlang records, but they're dynamic. I would very much like to see that. I'd also like to see a sort of separation between the specification of something and the implementation because we have hundreds of programming languages for programming things, we don't have any programming languages for describing protocols.

Do you mean network protocols?

Yes. I mean if you look at RFCs, they all define protocols in a very ad hoc manner, they never formally say "If I send you this package, you should send me one of these back". Something based on communicating sequential processes, Tony Hoare's old stuff, still very good in the background, when Erlang was first designed and really a closer correspondence between CSP would be good. Possibly some sort of notion of types, which is stronger than the dynamic type system we use would be good.

What do you mean by type?

I mean static types, Like Haskell's got. I think there could be subdomains where you can use static types.

Erlang is currently a dynamic type language. Are you saying you would like some static type as optional typing or both, or would you change the nature of Erlang to static typing?

No, I wouldn't change, but I would like subdomains where it is statically typed. You could actually encapsulate parts in statically typed.

Have you heard of Gilad Bracha's optional typing? Is it something like that?

I think the details would be different. If you could do it, there is no excuse not to do it in certain areas, but then you lose these lovely beautiful generic things you can do if you go static typing. I gave a course and there is a lovely example of - I create a server that does nothing at all - just an empty server. It's job is just to receive a message that contains code in it, and you just apply the code. I think it's funny, if you take something like Apache, you have a web server and you can plug in extended functionality, or you got an email server and you could plug it into it. Why don't you step back from that?

Just make a server that does nothing at all and put a plug in that turns it into a web server, put a plug in that turns it into an email server. A few years ago I built a generic infrastructure on Planet Lab, just put that in a server that didn't do anything, it's only ability was to receive code and execute that code. So "I don't know what I am going to do with this" and then "OK, let's turn it into a web server." And you send the code, and it becomes a web server. Later you say "I don't want to be a web server anymore, I want to be something else".

If we were to reinvent things all over again, if you want to get a file from a remote machine, why is the protocol for FTP completely different from the protocol for HTTP? Why don't we just have a simple protocol for getting and putting information? You only need one protocol, you don't need to reinvent it for every single application. In Erlang we have integrated this also, there is one way of doing messaging. On my lecture on Friday I've got a little example, which is ten lines of code - that's an email server, an http server, an instant messaging thing - it just integrates everything. Why can't you send the email to a web page. If I got a web page and I want to change that web page, why don't I send the email to it? Or why don't I send it an instant message? Why do we have these different worlds? I don't see this. It's a historical accident.

This is the same group that, when they tried to get them to swim together before or watch a movie together, they wouldn't do it?

Yes, I mean messaging everywhere with patterns just seems very natural.

You talked about types in Erlang, which brings me to another question. You have an Opinion on object oriented programming. That's an interesting thing to say in a conference like this.

You've been reading my OOP sucks blogs. I saw a blog yesterday, they said Erlang is actually more object oriented, truer to the spirit of pure object orientation than all object oriented languages. You could take two views: you could say either Erlang isn't object oriented - which I used to say a few years ago - or you could say it's more object oriented than all the object oriented languages. Alan Kay said that he made a mistake in not really emphasizing the messaging aspect of object oriented programming. He said in SmallTalk you talk about sending a message to an object, but you don't, you do a function call. I think function calls and remote procedure calls are too high-level in the sense that a function call or a remote procedure call returns to the person who made it.

If I do remote procedure call against you, I can say "Hey, do this!" and the answer automatically comes back to me, but that's not how we work. I could say "You do that" and reply to this guy. Why am I involved? That's how we behave. If you are a manager at office you say "Can you check this out? When you've done it, tell this guy". This is just an asynchronous message passing model. I think when you talk about object orientated programming, when you model a world, you are encouraged to think in terms of objects and I don't see objects. I also have what you would call the categorization problem: which object do I put my code in? Sorry I just don't get this. Let's suppose you have a method, in SmallTalk terms, that manipulates time of a directory structure or something like that, is this a subclass of a time object or is it a subclass of a directory object? I don't know.

Years ago I've started several projects staff - file my data/my letters, physical papers and research reports and things like that. Every time I've done that it failed, because I get files and I put something on this line - it says "Correspondence", "Lectures", "Papers". Then I get this thing: it's a paper, I put it in the file called papers; this is a letter which contains a paper "Do I put it in 'Correspondence' or do I put it in 'Lecture notes'?". It breaks down because I don't know where to put it. I think if you've got objects in your function, which object do you put it in? I made another filing scheme, in which I just numbered everything, and I have files and I go over my office and the first file says 1-167 and the next file says 168- something. I have a single file that goes 1,2,3,4 and some text. I use emacs to create, I use grep to search, and it's a brilliant system!

This is how we organize things in English. We have a dictionary and when we want to know what the word means, we look it up in the dictionary. Everything is in alphabetical order - that's a good way to organize data. I think object oriented programming is not a good way to organize data. There is a principle in programming "Never write something twice" - yes, it's a fine principle, a great principle, but let's think about this: "How do I find the old stuff, that I am going to reuse?" That becomes extremely difficult, in fact, I think people have been deluded into thinking that this choice is easy. When I started programming, there were no choices, I programmed everything in Fortran - Fortran was the only language I had on my machine. Nowadays, some poor sod's got to choose the programming language: "Should I have Java or C# or C++ or Lua or F# or Pascal or Prologue or Erlang or Haskell? On what should I do it?" "You don't want something in the-NET framework or a Java framework that runs on the JVM or on Mac OSX?"

You have all these decisions, incredibly difficult to make, and then you got bosses. You don't write this stuff in scratch! You got to reuse somebody else's code. After all that is. Somebody else's code doesn't do exactly what you want - it does something that's subtly different. Then, the time taken to modify and understand this old code takes an incredible amount of time. Have you ever sat there with code that somebody else has written and have to modify it? You spend hours and hours doing it. You can't reuse object oriented code, because all the stuff it inherited has to be there as well. You can't just take this stuff out and put it in your program cut and paste, like you did in C, or Erlang or Prologue or Perl, you got to take all the stuff it inherits from.

One of the problems I had with object oriented programming, one guy said: "You wanted a banana but what you got was a gorilla that was holding the banana and the entire jungle. You got a lot of stuff in it and all you wanted was just a little bit of code". The Lisp idea - lots of little functions - ubiquitous data structures like Lists, everywhere, and lots of very small functions that work on this, that these are highly reusable. Code that's referentially transparent, that means if you call it twice with the same arguments, you get the same value - it's very reusable. To them the question is "How do we reorganize all this stuff? I think something like a dictionary, like the English dictionary, put in alphabetical order. Don't put lots of modules, just put one module! If code is referentially transparent, it's small functions, then it doesn't belong to object. Where does it belong? What is the containing object? There isn't one! So, just put them all in alphabetical order and use Google to search it and you'll find the code easily.

I'd like to talk about memory management in Erlang a little bit. Every process in Erlang has its own memory. How is garbage collection? Is Erlang garbage collected or reference counted?

Absolutely, it's garbage collected. There have been different implementations that have been reference counted, so it doesn't really make any difference. The user is not supposed to know what a garbage collection is. Each process garbage collects its own memory. It's not strictly real time, but because processes are small, the garbage collection time from individual processes is very quick, so each process has its own stack and heap.

That's the big advantage compared to languages like Java which have big shared heaps, where garbage process will be very long, it can be very long?

Yes, I mean it was written with soft real time behavior. It's extremely unusual that Erlang programs are bothered by garbage collection issues. An Erlang programmer has to be aware of the fact that if they write their application as one gigantic Erlang process, the garbage collector might impinge from the other processes slightly, but it's usually not that. It's so rare that I don't think it occurs in practice. Theoretically, it should occur, but it doesn't seem to. It's not an issue, really. And then of course you want to have copied all the data.

It didn't make any difference about 10 years ago but when you go to multi cores, the advantage of having a complete copy of the data, you don't need to lock anything anymore, so everything just runs asynchronously, you got all you need to do your computations. If you have shared memory you have to coordinate and lock it and, of course, when you lock on a multicore you have to just stop all these processes while you are gaining the lock. I don't understand how that works actually. I know how Test-and-set works on one processor, but how do you generalize that? You stop them all, it's a super big test Test-and-set while you are doing things - it's very confusing what happens.

Recent implementations of Erlang are now multithreaded, they can use a sort of m:n model of threading, is that right? Did you use processes where they usually use userspace threads?

No, I think the current implementation takes 2 threads per core - one thread for Erlang, one for IO, because IO is asynchronous. As far as the operating system in concerned it's taking one process with 2 threads in it, although inside we are emulating everything that the operating system does. We got run queues and schedulers and message passing and garbage collection. Basically Erlang just asks the operating system for a big chunk of memory. Of course, it uses the operating system to give it file descriptors and sockets and things like that, but the operating system is just like a device driver for sockets and file descriptors and TCP and stuff like that.

You mentioned asynchronous IO. How do you do IO? Do you use non-blocking IO?

Yes, in Erlang you have a port, you send messages and you receive messages from it and this is non-blocking and all the other processes just proceed while it's waiting. It's like a fancy select and it doesn't do any busy-waiting or anything else - it's pretty efficient.

Another thing that I found interesting in Erlang is how it communicates with other languages. I think unlike languages like Java or so that can load libraries, I think Erlang doesn't do that?

No, it doesn't and that's deliberate. You know, there is an easy and a difficult way to connect components together, and the easy way - the prime example is the Unix pipe mechanism: you say "cat this file, pipe that, pipe that, pipe that" all written in different languages - no problem at all! It's completely trivial to connect the applications together. There is a difficult way of doing that and that is the link to memory space using API - it's appallingly difficult. Everybody uses a difficult method of connecting stuff, not the easy method, because it creates jobs for programmers or whatever. Now, the trouble with linking stuff into the same memory space is the accidental side effects of errors.

If you link somebody else's code into your memory space and that code crashes, it crashes your application. Erlang is built for fault tolerant systems and therefore it does not allow you to link anything into the same memory space. That's the fundamental reason why you are not allowed to link stuff into your memory space. That means you have to go through an IO barrier, so you have to serialize everything with a socket and run it outside Erlang - that's the reason, because people do not build libraries with the intention of running against the socket and serializing things. But on Erlang, once you've done this exercise of serializing everything, it means it's network ready. You can run your Erlang over a network and because it's talking a few sockets, you can run your application on one machine and the foreign application that's running on another machine just talk. Once you've done this exercise of serializing everything, then it becomes much easier to connect things together, but there is an additional effort in serializing things.

I just think it's absurd how things are connected together. I've got some midi synthesizers at home. Why don't I just send a UDP packet? I should have a port called Violin and I'll send the UDP packet that just contains play notes, G# "loud" and then "Play". I've done music on an Apple. Sorry Mr. Apple, but your API sucks. I've got this incredibly complicated API, with hundreds of things and 3 or 4 things are going to be connected in exactly the right order. If that program crashes, it crashes any program that links its code into it. It's a stupid way of connecting things together. I think you need to go back to this philosophy of Unix pipes. When you go to a pipe, if you do "cat foo ?" you are sending character over this boundary. In Erlang you are sending structured terms over the boundary, so if you say "a#{b,c}", then on this side of the boundary you receive "x" a tuple.

So, you've got this kind of built in parsing that doesn't occur over a Unix socket or character stream. Given that your processes just exchange Erlang terms, just piping them together becomes a trivial exercise. The first thing you do when you write an Erlang program - you abstract the behavior of … something into Erlang messages. For example, if you write a web server, you put an interface here and you say "get"- this is an Erlang term- and out comes something horrible "http1.0". This is the actual protocol you see on the wire. You put a device driver in between and that means you could write lots of different device drivers, so this could go out to be an FTP. Remember I said "What's the difference between FTP and HTTP?" You can abstract that in the device driver so you still see a uniformed message here. You could use XML or any structured type across a boundary and you could automatically parse it, but if you used XML you'd be affected 20 times as inefficient as using Erlang, just because XML is a very verbose format. If you have a universal messaging syntax it becomes very easy to connect things together.

Is there another way to connect to other languages, like a device driver?

Yes, there is. If, despite all the warnings "Thou shall not link other people's code into your code because it will crash you code if it's wrong", if you wish to totally ignore that and make it very efficient, you can actually link foreign code into the same memory space. Then you can crash all Erlang processes. Say you have a a million of processes, so suppose 5,000 of these want graphics and you crash - there is some mistake you've linked in, some graphics package, because 5,000 programmers or processes wanted to do graphics, you crash all the other 900,000 processes who didn't want graphics. Remember I said Erlang is much more like an operating system then a language? I think the analogy would be - you'll understand why it's very dangerous to link a graphics package into the Unix kernel, you are putting it straight into the kernel.

That means, if you make any mistakes, you crash everything. That's not what happens in programming languages, because you are only running one application, so it doesn't matter! If you have this model, it will only going to run it once, then it's fine to link it in foreign code, because you crash your application, but you don't crash millions of other instances of that application. If you want millions of instances applications, you have to use processes - that's what processes are for, but you can't do millions of them. On a normal operating system you can do 2,000 processes, maybe 20,000 if you are lucky, which takes lots and lots of memory, because a process is very big. If you want 20,000 instances of your thing, you need 20,000 processes and each one runs one instance. If that crashes, it doesn't affect the others. If you put them in the same memory space, then you crash everything else if something goes wrong. Erlang started off for telephony, a telephone switch might have 200,000 things going on, conversations going on. You don't want to crash all of them if there is a mistake in one of them, so you need a memory protection mechanism - that's what Erlang does at a very fundamental level in the language.

I guess the name device driver is quite fitting as compared to the systems? Does you also see the danger of writing it?

Yes. It's very dangerous, but some things are done with these linked in drivers and they have to be very well tested.

Do you mean file system, access?

Yes, things like that. But they have to be very well tested, when you do this, the guarantees are off. Don't do this at home because it's dangerous.

Erlang is built on a VM or is a VM. Is it an interpreted VM, byte code interpreted or was it used not currently?

It's a virtual register machine, which is interpreted and it can also be compiled. I mean there is a native code compiler.

Is this HiPE?

Yes, this is high performance Erlang, done at the university of Upsala. It compiles to virtual register machine instructions, which are then either interpreted or cross-compiled to native code.

Is there a reason why the Erlang VM is a register machine?

Yes, it's better than a stack machine. First of all, how do you make a register machine? -Usually first of all you make a stack machine and top of stack is always registered, you spill it, you say "Well, I'll mirror the first 4-5 elements on the stack and fixed registers and then the others. I will use a stack anyway to spill them off." It's just a few stack counting the number of memory fetches and stores and memory accesses. It's better to have a register machine than a stack machine.

Is it an infinite register machine or is there a fixed amount of registers?

Infinite? - I don't think so. Because infinite is very big.

I think there is a virtual machine called Parrot which I think it has infinite amount of registers.

I wonder how they do that. I think of my turn out as a finite number. Because infinite is really big. We don't do infinite precision integers, we just do them with billion of digits if you want, but not infinite.