Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews Joe Walker on Bespin

Joe Walker on Bespin


1. We are at QCon London 2010 sitting here with Joe Walker. So Joe, who are you?

I work for Mozilla, work for Mozilla Labs really which is a part of Mozilla where we try out new stuff. I have worked for Mozilla for little over a year now; I spent most of my time working on Bespin. Before that I spent most of my time working on DWR which is a framework for doing Ajax with Java, so I have probably been working on that for four years or three years or so before I went to Mozilla, and now I spend most of my time on Bespin.


2. What is this Bespin thing?

It's a cloud editor I suppose. So the name, for Star Wars nerds, is from the planet in the cloud from Empire Strikes Back. So it's a cloud editor the idea is primarily that you can store your files in the cloud, you don't have to install a big heavy weight system on your desktop you can go home, you can move around and you can have the same files wherever you go. That's the top level summary I guess.


3. So what does 'cloud editor' mean specifically? What technologies do you use to build it?

It is built predominantly using JavaScript, the majority I would say seventy percent of the code is client side code, JavaScript. The server component is currently Python; we have definite plans to change that into being JavaScript, so it's JavaScript from one end to the other. [Editor's note: the current plans for using Node.js on the server are detailed here:


4. What kind of server for JavaScript do you think you want to use for that?

It should be some sort of CommonJS platform we are actually using CommonJS in the client as well. The idea is there will be some components that sometimes it makes sense to run them on the server and sometimes it makes sense to run them on the client. So a great example I suppose would be if you have got some function that need to look into all of your source code and you have got significant projects, you might want to do that on the server because you got local access to all the files, you don't have to transmit the entire project across the wire in order to do a search across all your files for example, that's something you probably want to do on the server. All the same there will be search functions; do you want to do that on the client as well? So search is kind of both sides. So there will be stuff that we want to do in both places and it makes sense if we don't have to re-write that all the time.


5. Bespin has been around for about a year I think and you recently rebooted the development, what does that mean?

I have been on it for about a year it's been worked on for probably about eighteen months so for the first year of its life the first six months of the time I worked on it roughly speaking, it was an experiment and the question is "Can we do this? Is this viable?" And about six months ago roughly speaking, we said "Ok, there is a lot of this that is viable" and we have been doing it in terms of what can we do? Features, features, features and that can only go on for so long, we need to sit down and say let's learn from what we have done, let's do some architecture behind it and let's take a step back and think about where we are going and re-organize things a bit. So that is all we've been doing, in the past six months we have been just going around, refactoring it I guess pulling it all into line, making sure there's is a layered thing going on, that sort of stuff. Now the reboot sort of comes in two phases: the first phase is rebooting the client, and the second phase is rebooting the server. The server is currently written in Python and when we started the options basically were we mucked around with Java and Python, and we stuck with Python and when we reboot the server which should be the next phase of the reboot, we are likely to rewrite the Python in JavaScript. There is not huge amount on the server at the moment, it's around thirty percent, it may even be twenty percent, but that would be the far smaller chunk of work.


6. So you had a protocol essentially to talk to a server and to write other projects like Eclipse that wrote backends for that. Is that something you encourage?

Absolutely, in fact there have been a number of backends that have already been written, we had a PHP backend, we had a Python backend, we've had a Java backend. In fact we had two Java back-ends: one Java backend was the original one that Ben wrote, and then the Eclipse guys came and wrote theirs, perhaps using some of Ben's code. So a bunch of people have worked on different back-ends, we totally would encourage that sort of thing I think we are likely to focus in the future on JavaScript. The interface between the client and the server is a REST API published, fairly simple not actually hard to reimplement.


7. Moving on to the client: how is Bespin implemented on the client? How do you use JavaScript, how do you make it work?

Ok, so we have got a MVC architecture, the view at the moment, the editor component is a canvas and the command line components and most of the other stuff is plain HTML 5. The reason we chose canvas is simply from a performance point of view I think we are quite keen that the editor is quick, from my point of view my biggest gripe with big IDEs is how often you want to type something and it is doing something else and it is slowing you down. So speed I think is a big issue. Canvas has got the performance in spades. I am keen that canvas is one of the views, not the only view, because there are downsides to canvas: the accessibility on canvas is in my opinion it's not worth us trying to make the canvas accessible. And we are much better off writing a DOM based interface, which is accessible. And we may well end up writing two, we'll see. But the thing is definitely designed for having more than one way of displaying it.


8. Would a DOM based interface be possible, speed wise, would it be slower?

Yes, certainly there'll be nothing to stop you in theory just having a text area and dropping a big file in it, that would work. It starts to fall down with decent sized files pretty quickly, and there are a bunch of other limitations with text areas. One of the tricks we're doing with canvas for example is to only repaint the bits of the thing we need to repaint; that is one of the problems with just taking a text area and dropping in all of the text in it, is the DOM has to cope with the whole thing. You could actually pull the same trick with the DOM, you could have yourself, you could detach the scroll bar from the rendered area, maybe you've got an 80 row display, twenty four column display it is, then you could arrange for the scroll bar to be a totally separate thing somehow, and you watch what the scroll bar is doing and you poke into the right lines, where the text is, you can pull the same tricks. And there are plenty of GUI toolkits that do exactly that when they are re-drawing stuff. There is still a speed issue, they are quick but perhaps not as quick as they could be. But my expectation is that that kind of approach will be viable before too long. And maybe there are other tricks we can pull as well, but the reason for using canvas is kind of pragmatic right now, we want to get the thing working, and looking nice, and I think there is a bunch of experiments we would do once we've got the thing working, get it working first and then do other stuff.


9. Does canvas have all the features right now that you need or is there anything lacking, any type setting features or things like that?

No, I think most of the restrictions are kind of implementation things. So one of the tricky questions: typically a text editor uses a mono spaced font. As soon as you use a mono spaced font you are making a kind of a western bias because a number of character sets have characters that can't be mono spaced. It just doesn't make sense. Now there are some good performance benefits, from having mono spaced font because it is a simple sum, how long is the line? Well a number of characters times width of characters equals length of line, it's easy. If you've got a variable width font, all of a sudden how long is a line becomes a hard question. You click somewhere, where did he just click? It's no longer an easy sum. So we currently made that simplification, it's a mono spaced font, and the majority of editors have done the same thing. We would like to remove that restriction, but we need to see what we'll tackle first and I think we'd rather make it usable now and add stuff in later, as in when we can.


10. Looking at the architecture of the client Bespin, how can you extend it? How do you build it? Is it monolithic, or is there some plug-in system?

Yes, so the entire Bespin is a set of plug-ins with the exception of the very small layer at the bottom which makes the plug-ins work. But the editor, the command line, the collaboration, the model is a plug-in and so on. The whole thing is plug-ins.


11. So at what level of abstractions are these plug-ins? What granularity? Is it big features, is the whole GUI plug-in or how does that work?

They vary in size, I mean some of the plug-ins are really quite small. Then one of the most basic plug-in is the types plug-in for example which just provides you with some basic type like string and some methods on those types like be able to validate a string convert whatever it is to some serializable forms and some basic stuff. That is a fairly small plug-in and the core editor is a component is a plug-in, so they vary in size quite a bit, I don't think we want to aim for a plug-in should be x lines long, because it's kind of a logical thing, we haven't really spent a lot of time debating the size of the plug-ins, they just kind of happen naturally. That's been fairly easy. One of the nice bits about everything being a plug-in is that it is all re-loadable dynamically. I mean we can edit Bespin using Bespin when you hit save and Bespin notes that this is it, it is itself if you see what I mean, so you can make changes and you don't have to hit F5 to see the effects of your changes.


13. It's a bit like last night's talk by Dan Ingalls...

The Lively Kernel thing, yes. I mean we've come in a higher level, we are not Kernel, we have not attempted to build the world up from trivial graphics primitives, we are writing the canvas and the DOM, that sort of thing. So we've got a big head start and there is lot of flexibility, text is not rotatable like in the Lively Kernel. There are reasons for doing it different ways: their primarily goal isn't to build an IDE whereas ours is, so there is reasons for both.


14. Bespin is supposed to be an IDE so can plug-ins hook into the whole system of editing? Can they provide styling in the editor or add markers or something like that, how does that work?

Yes, so the whole syntax highlighting thing is a plug-in and provides some extension points itself, for different syntax highlighting systems. So if you wanted you could take out the current syntax highlighter and plug in a new one because that just writes to the model, part to the model about how things are highlighted, and there is a way of saying, sort of based on regex expressions, to decide a context and how to highlight a context. And then there are plug-ins that are built on that to say "Ok, this is how to highlight HTML, this is how to highlight a CSS" so there is sort of a hierarchy of plug-ins all sort of working together to create something and you can rip out the editor if you want, replace your own editor but then of course you got to provide your own hooks to syntax highlighting and syntax highlighting engines and the actual ways of..., this is a kind of a layered thing going on there.


15. So it is very extensible?

Any of those bits anyone can replace. The tops of the tree are more likely to be replaced than the bottom of the tree but any of that you can hack about with.


16. So essentially if you edit a file in the editor and you have a lot of plug-ins listening to edit events, how do you keep the editor responsive?

There are two answers to that question: the primary obvious thing is to make it asynchronous. The current syntax highlighting is asynchronous. So if you have got a very large file which is an HTML file starting with <html> and you got bunch of plain HTML and the you go to the very top and then you just type <script> that changes everything the entire or the rest of the file, it has to go and have a look, a re-look at all of that and it's probably completely invalid as far as <script> goes but the point is that a single character may have a significant effect on the rest of the file. So yes, it's an asynchronous thing going on, where you decide that you need to go and re-parse some area so it asynchronously goes off and does that and then goes back to you with the answer. That still affects the main JavaScript thread as you are typing you have to interact with the syntax highlighting system. We talked about the reboot and old Bespin was the stuff before the reboot, in old Bespin we were using web workers to get that off the main thread. We may bring that back at some point but the moment it's fast enough. Performance is very definitely a prime goal, it should always be as fast as any native editor.


17. How do you interact with Bespin? I see you have a command line. How does that work?

I think we see a lot of the complexity involved with big IDEs is a lot of windows and stuff around the edge. They can become quite cluttered. Sometimes the clutter is you need some plug-in to do some little thing but it ends up having its own menu component and its buttons and stuff everywhere. My IDE at home has a source refactor menu which kind of gets duplicated and I have to go and turn these duplicated menu items off otherwise I get a ton of them, contributions by different plug-ins. Vi, TextMate, Emacs, have all got command lines and there is a lot to be said for command lines in terms of speed, in terms of minimalism and simplicity. So we like the idea of a command line, there is a down side as well they are fairly opaque. The advantage of a GUI you can look at it and you can tell what you can do with it often. With a command line you just completely don't.

Our plan really is to enhance a command line so it's not as opaque as it could be. I mean really command lines we have got this kind of thing that command lines are character based. And there is no good reason, why do we all assume that command lines have been purely character based? Because that is nonsense they don't. If you want a thing to run on a Linux console maybe it has to be character based. But if you are on a traditional console or telnet or something then the character assumption is nonsense. We are using HTML obviously in the output which gives you a lot of options. And one of the big things I suppose is that the output can become interactive, you can if you have a complicated command, a great example would be if you are doing a checkout, you are checking some stuff out of some source control system now if you want you can type that stuff on the command line or you can just say checkout return. And what comes up in the command line is a little form to say where are you checking from, what are your credentials, what is the version control system, etc. you can type that on the command line or you can fill it in. I say that is a great example of just breaking out to the mind set of character based UI. One of the other things is command lines give you completion but that is the only help they give you in typing stuff out. They ought to be able to, as you are typing stuff, tell you, you type a command and up pops a little man page summary of the command. There is a bunch of stuff like that which you take away from the opacity of the command lines to make them discoverable like a UI is. And our hope is that we can have the best of both worlds. We can have something that is fast, minimalist, and yet not as opaque and hard to use.


18. How do you deal with the problem of namespacing of, if you have a command line you have to find a way to find names for commands but do you have some namespace feature for that?

No we don't, I mean we have a concept of subcommands, this is something that we are familiar with from version control, so every single version control system pretty much has a basic command, CVS, SVN, HG, p4, git, once git got its act together and then everything else is a command of that. So that is kind of a namespacing thing and we support that. And there were bits of Bespin that is a similar system so that is sort of namespacing I guess. We don't have some complicated Java-esque reverse domain name thing going on that would be a bit slow, a bit annoying.


19. One of the features of Bespin is collaboration. What does that mean in the context of Bespin?

One of the advantages of putting the thing on the web is that you have much easier access to the other people using the same system as you. One of the things about software development is however much we may think we are working alone, even the guy sat in a little box writing code is using stuff that other people wrote. So other people are a far more important part of software development than I think we often realize. Pair programming is an obvious way in which there's a direct interaction. And old Bespin supported pair programming. It was a bit hacky in places but it worked and you were able to have lots of people working on very large files and it was ok. So we just bring that into the new Bespin, into the rebooted thing, there is a bunch of things that we want to do on top of that I guess. It would be nice if you and I are working on the same project from different parts of the world and yet we probably barely know each other. If other people working on the same project as you, you might not know anything about them but if it's your project and I am proposing some change, it may be good for me to know what you have changed since you last checked stuff in because I want to avoid those lines because when I am going to check stuff in I am going to have to merge.

Also there is this concept of what you are doing, which crops up in a whole bunch of places, what am I doing crops up, you might have it as a status message, kind of Twitter-esque thing, but it crops in commit messages, it feeds into your bug tracking system because you have information about what you are doing now in your bug tracking system I am fixing some bug or other, it's in commit messages, chat, all that kind of thing. There is a simple concept of 'What am I doing now?' that crops up in all kinds of places; all of those systems are about interaction in some way. So I think one of the things that we would like to get right is having this concept of an IDE - most of the IDEs traditionally tend to be a bunch of tools bolted together. And because we are sort of rewriting everything I think we have the opportunity to do a deeper integration. And so previously there was no centralized concept of what you are doing now, across your IDE, you could just tell your IDE what you are doing now and when you next came to check stuff in it said "Well, when you were working on this file, you told us that you were doing this, that and the other. Here is your commit message automatic". So I think some of the integration that we could do, there is social integration that brings stuff together and help the interaction which is I think quite important to have everyone remember that we are not working alone. We are all standing on the shoulders of giants etc.


20. Do you support real time collaboration in that sense in the editor?

Absolutely, yes, old Bespin did that and new Bespin will do soon hopefully.


21. Bespin has been around for a while and we have already seen third parties adopting it in their products. Can you talk about some examples and how they customized Bespin for their products?

I suppose the most obvious example is Palm who have taken Bespin and made it part of the development environment for Palm Pre etc. Currently they are just moving over to the new Bespin I think that is the plan anyway, and basically they have taken a whole bunch of customization, they built stuff around it, they have taken basically the text editing components and built something that looks perhaps more traditional IDE like, which is certainly possible. With the new Bespin that would be a lot easier because it's plug in based, you don't have to go around hacking things up. That is certainly a way that you can hack it up. Another example would be just as a little side project, some of the guys at Meebo are doing; so Meebo is a web chat system so it would take your MSN credentials basically give you a web page. I guess the primary use is getting around firewalls, they are using Bespin and they have hacked up a local file interface backend into it and again, old Bespin, but we are just getting to the point where they're starting to hack on the new Bespin. And certainly the idea of replacing the backend is a key thing.

There are bunch of ways of doing it that make sense, so it makes sense to have some remote server the kind that everything is hanging off. It makes sense to have a distributed set of servers; traditionally I guess that is more like how development works, you have got your IDE and it talks to a bunch of version control systems for example. So it makes sense perhaps for that kind of interaction. It makes sense to do the traditional working with the file system kind of approach and we need some kind of browser plug-in in one form or another to do that. In theory there is no reason why you couldn't install the server locally, talk to the server and it talks to the local file system and there are some advantages of that, in theory why not even store your files in the browser's local storage area, no plug-in required obviously you can't get to the file anymore but everything is local. I mean each of those is a different back-end. So it makes total sense to have a pluggable backend system and we are concentrating on the remote server for the moment but all of those other examples will pop out and others than we haven't thought of yet.

Jun 21, 2010