OK, I actually discovered Ruby in the spring of 2001, coming out of doing about 3050 lines of Java code and never wanted to touch the language again. So I needed something different and I started looking at Python as a language and was very interested in it, and then I found the Pickaxe book, Dave and Andy's book. And I read that front to back in a weekend, and I just fell in love with the language. And so, when they offered the Ruby conference in October that year in Florida, I went to that, I was actually between jobs at the time, so that was on my own nickel as a lot of people there were, it was a hobby, but I really loved the language. The first thing I did with Ruby is that I actually ported the Java library that I wrote, in Java to Ruby, so use Java for a library. Then I did some more experimentation with it and I realized this could be a great language to build applications with. This was 2001 so I tried to do a start up with that in mind. But with the 9/11 happenings and everything else, start ups the 2001 were'nt going anywhere.
2. Probably the wrong time to start something
Yes, it was really bad. What I did is I hired back the core group of developers at my Perl company, where we did all this Java development, and we actually got a gig at Darpa, doing research for Darpa, really we were system integrators, but we were all architect level developers but we were doing systems integration. And we said "we will do this but we will be writing code to automate everything". And they said "great". They had about 30 machines when we got there, there was a distributive multi-agent system, and we wanted to basically use Ruby to build test harnesses for the whole distributive multi-agent system. That was all written in Java but we used Ruby as an Agile framework to test a very complex distributive system. By the time we left Ruby was used ... to the program. We built a test harness; we had 350 machines by the time we left, dual-xeons in racks. And Ruby controlled every single one of those machines. We could rebuild the machines from scratch with Linux using Ruby. We used Java for a messaging system to control all 350 machines. To describe the multi-agent system we used Ruby, everyone in the project was actually using Ruby to configure everything, but we used DSLs to do that. To them, they were just writing a domain-specific language that looked like their domain which was logistics.
3. Were the DSL and internal DSL just actually Ruby code that happened to be?
Yes, exactly. It was really funny because as people were doing this they were like "What if I want to do this 3 times in a row?" 3 dot times do. And they all thought I was brilliant, of course it wasn't me that matched it, but it was great. And we used that very successfully, that Darpa project was successful.
4. DSLs and Ruby are not a new thing.
No. We were using them and they weren't really talked about then, but it really seamed natural, because the language was so flexible, and I was the one that came with the "3 dot days plus 5 dot hours" because I was trying to describe time in this thing, and they picked it up in Rails, and they have that in Rails and a lot of other things. From there I actually got into the Semantic Web project and the first thing I wanted to do, I was looking at that project and I wanted to write an OWL parser for Ruby, ontological web language. I wrote something called Semitar in Ruby, which did that. It basically worked with the OWL object model in Ruby and I presented this to the guys on the Semantic Web project, and they were all blown away because they were doing all that stuff in Java and I was doing this in Ruby but I tied in the raptor parser which was written in C.
I did! This is on the demo project, the Darpa project. They had these databases which were several hundred thousand RDF triples. And they were doing everything in Java, and I came along with my Ruby framework and they said "Well, you can not do this with an Agile language, this is not going to work with a dynamic language, you need something statically compiled, we have huge data sets.
They told me that. And I said "watch me". So I tied in raptor which was written in C, to parse the RDF triples, but actually manipulate the RDF triples and it's not cheating, that's what Ruby is about, it's a scripting language. So I showed them this, and of course there were Ruby jams at the time, and I showed them how Ruby is running, my Ruby parsing, actually a triples query which I wrote in Ruby. It was faster than Java, and I was about twice as fast as the comparable Java library which they were blown away by, and then they started using it and it actually worked with their triples. And I solved the problem of implementing the ontology in it but the reason I think that stuff is really going anywhere is Descriptive Logics and OWL are really complicated, you get into it and trying to understand it it's not simple. It's one thing to take HTML and build a markup of a document, it's another thing to try and semantically describe things.
7. It's a skill; it takes a lot of study...
Exactly, and there are a lot of people who are excellent in building anthologies, but I think that that whole domain is new to most developers.
8. Do you agree that maybe it's just something that is ahead of its time?
I think the issue is that the people that are working in that field are traditionally from the AI field, they aren't corporate developers. The corporate developers are building systems over here and these people are using these things to build knowledge bases and query them.
No, they are doing research; they are getting paid to do research. And then the people over here try to build these systems. Now what I think can actually be very flexible, is start getting into systems that need to change more dynamically, larger systems that need to change dynamically. Ontologies can be use to allow things to be dynamically composed. And I know people are doing work in that space.
Right. I lost interest in a lot of these things. Another library along the way that we wrote out at Darpa, at the time we had about 300 Java processes being, one agent society was composed of 300 Java processes running concurrently. And they wanted to be able to freeze these 300 JVM's at once. And there is a really cool thing called the Java debug wire protocol or the Java debugging interface.
11. And it talks directly to the virtual machine
It does, under the covers. And it doesn't use Java serialization, it just uses binary data going back and forth, so Chad Fowler and I over basically 2 weeks, took the Java debug wire protocol specification which was an HTML document on their site, turned it into DSL in Ruby, used that DSL which was the spec to generate the packets for send and receiving data. So we used the DSL in Ruby as a generator to generate Ruby code, as the whole protocol and then I used that at Darpa. They were trying to say "we could freeze the agent society from within the agent society will send messages, and it took about 7-8 minutes for all the messages to propagate and everything to freeze and then go quiet. I had a Ruby process that was running all 300 VM's were underneath it, and I could freeze it in about a half of second. All 300 of them. And you actually could watch the CPU use because we had a monitor and the CPU use has dropped to zero. And it freaked them out. And what was great was you could turn it back on, and all the agents came back on. But time had been lost. And it was like an alien abduction lost time, ten minutes went away, "what happened to us?" It was a bizarre thing because they were agents and they were planning on things and all of sudden 10 minutes just went away. But it was interesting to show how Ruby could actually be used as this kind of harnesses to wrap around things like systems.
12. I've heard it described as soft layers and hard layers
Exactly, and Java, C sharp, a lot of these things are very good at building hard layers. But the soft layers are absolutely necessary. As a matter of fact when you composite things I think the soft layer compositing would be far better than trying to do hard layers... They are trying to bulk hard layers together but there is noting in between.
13. And I love the catch phrase "Ruby is enterprise glue that doesn't set"
That's exactly right. If you think of it as a vehicle, how much gaskets you have inside that vehicle. Because you don't want the things rubbing against each other. But right now, because people are so against learning new languages, which is really sad, but because they are against learning new languages or getting new languages into a development environment, they want everything to be in Java, everything to be in C sharp, everything to be in whatever language they are building in, instead of saying: "let's use different languages for different pieces of this", which makes sense.
14. Is it time to change or do we just have our Ruby color glasses on?
Rails has helped, Rails has shown that it's the good that doesn't set. But it's actually an application, the hard part of it is the database, and everything else is soft, everything else is dynamic and can change and it let you actually build that system. But at that point the database is the thing that is the robust scale for enterprise thing, and you have all these other elements, the Rails parts of it which are very flexible, and it lets you build applications very quickly in a scaled application, and it works.
15. Were you the only Ruby lunatic running around Darpa or are they using Ruby a lot now?
On our project I was the one that pushed it all in into the project, by the end all 100 people in our project were using Ruby just not overly learning it. But some people from that project they went on, I was across a lot of different companies bowing ...company that went on and used it in other ways. I know the guys at BBN they were our prime contractor, BBN Technologies, there's a lot of people at BBN now that are using it. They are using Ruby up there.
16. So there's a lot of Ruby adoption under the covers that we don't see?
These people are scientists; they have 800 scientists that work at BBN and doing a lot of different things. I met this one guy, brilliant guy, Ph. D and his focus was logistics. He understood logistics inside and out, and he had all this data in spread sheets, for all this logistics data and then he had to go to the Java people, to implement this stuff and actually move it into MySQL database where they could actually define their agents. This is an example for doing that. And I told him "you need ... data in these files, why don't we dump them into a CSV file". So we did that and I set with him for one afternoon and what we did is build a DSL, a Ruby object module of an agent society. In an afternoon he got all his CSV to where he basically had his entire thing that used to be in a spread sheet in a Ruby source file that he could read and made sense. So it was readable to him it was more readable than that CSV file. And from there we said "ok, if you define all these organizations this way, I bet we can figure out algorhythms for what plug ins go with what agents what behaviors go with what agents, based on the organization" and he said "this might be possible". So within 2 weeks he had basically taken something that I wrote to give him some infrastructure for that, and actually built all the rules. They used to be in MySQL and the Java guys were no longer involved.
Another example at BBN actually, I get last fall, was a mapping application, with the airforce. And they called me up and said: "we've got this problem. We've been working on this project for three years. We've been trying to build a functional prototype". The project was to build a user interface specification, for a design that had 13 different organizations that needed a shared experience. They are human HCI specialists, brilliant people doing all that, interacting with all these different organizations, it was for mid-air refueing of aircrafts. That was the application. And when they actually have these airplanes go from one point to another and they are refueled in the air, it comes done to the point where if these planes aren't all refilled within a 5 minutes window, they all abort. And planning those missions out is very complex.
They need a bunch of different systems, but nothing that was a unified shared experience. And so they have been working on this for 3 years to interview all these people, design the shared experience but they wanted to build a functional prototype of the interface. Great idea. So that the specifications would come with the functional prototype wouldn't be used in production, but it would show what they meant by the spec. Again, very good idea. They tried to build it in Java, and they had 3 people working for 18 months on this and the example app they built, the functional prototype kept failing. I was asked to come in and basically I had one man-month, myself, to build something. So I done a bunch of work for this other project that we are working on now, with Ruby and Flash, to tie those 2 things together so I can use Flash for user interface and Ruby as runtime and I said "if I do this, I want to use the tools I want to use, to build a functional prototype". They said: "as long as at the end it runs on Windows, I'm fine" and I said "well, do you want better? It will run on Windows OS10 it will be a double click executable" and they said "ok, we're in". So what I did is I read their documents that they have created and I built DSLs to represent all the domain objects. The first 2 weeks of my man-month were spent doing that, which they thought was a little crazy.
18. But you don't worry about the implementation until later.
That's correct. As a matter of fact, the first day I worked with them I spent the whole day writing the Ruby syntax. And it was hilarious because I'm working with this woman, brilliant person, Beth ..., she's and HCI specialist, we were working through the domain objects, and I said "this is the way she did this", and she said "yes" so I'm typing Ruby into an editor, and she said "what's that colon in front of that thing" and I said "that's a symbol, that's why it turns red because it's a name of something", and then as we were going through it she said "we need a symbol there". So she was even speaking the proper Ruby terms and it was hilarious. But she and actually the rest of the people on that project, which were all HCI folks loved the domain model, because what happened was I found logical errors in their spec. Because once you actually have to build something out and run it, then the spec validates or doesn't. We found logical inconsistency, they had to go back to users they talk to in two years and this was all within a month. Well it was 2 months but it was a men month of effort but I had 2 months to do it. But I built this whole thing a simulation that actually drove a mid-air refueling with a map I actually build a mercator-projection map in Flash
19. There's some graphic stuff that you had to do.
Yes, a lot of graphic stuff, it was actually more graphic stuff than Ruby stuff. But the Ruby model drove the Flash interface and the simulator was built in Ruby. The simulator actually simulates the plains moving and everything else in 30 lines of Ruby code. Walking this object model they just changed the object attributes and the domain model. I didn't have a database I was like "I'm going to have Ruby source be the database". And it worked, and we delivered the functional prototype and the project was successful. I got paid real good money, because it was a month of effort, but it was great and what was cool though, as a partner BBN is a great partner. And it was great that the project was successful. But the arguments I heard from folks that have built the Java thing was "this functional prototype we built could be deployed" and I said "well, it's a functional prototype, you don't want to deploy it" and they said "but we can". It's a very strange thing, people that do nothing but build prototypes or research every programmer wants their stuff to be used. And when they don't get it used, it's a let down. And they pour all this effort trying to build something which could be used, and sometimes it is and unfortunately it's a really bad thing. When you're building a prototype you don't necessarily want the prototype to go to production, and I was asked: "why doesn't this go to production, this is great". And I said "it's prototype -- you're not going to go into production with Flash and Ruby". Now... they could have!
No, and I think in that scenario, that's absolutely right, I mean here there were so many things that the airforce required for mapping their C++ framework in ... All these systems that you have to touch, and integrate in, this was literally a functional prototype of an interface. And it's that, and it works, as a matter of fact we put it on a CD, double clicked the executable on a CD that could run it, it was all Ruby driven, I even documented how they could change the model to actually see the system behave differently, where the interface deviated from the spec, which it did a little bit, I couldn't do it perfectly, we documented that, but almost everything in the spec was in the interface and so it gave them a concrete example, the implementation is probably going to take 4 or 5 years, to build the system. That's fine, but at least they know what the interface is going to look like
21. I'm going to give you a chance to plug Indi, because it's working on that I'm sure helped you out
It's actually describing what I used.
Indi came about with the idea of once the Darps project were over I think we want to go to product ... what kind of product will we do. And we have basically 3 developers, myself, so 4 developers total, and the C0O, business guy, so the 5 of us in this company, and we try to decide what to do. What we came up with was the idea of "what if we could build a really small PDA? About the cheapest PDA you could possibly think of, what would it look like?" And we ended up like a USB Flash drive. USB Flash drive is portable storage, it's like a hard drive, I can plug it in and use computing equipment all over the Earth through this thing and it works on Windows or OS10. So it hits all major platforms, but you need a software environment on that device, so that when you plugged it in you used the screen of that computer, you used the keyboard of that computer, but all the data was on the drive and all the data needed by definition to be encrypted or else if you lost that drive you lost all your data. So Indie started with this, started with "we're going to build a portable information environment that ran across platforms". Ruby gave us I knew I wanted to do the one time in Ruby. Because I had the data model that we built which was very flexible, scaled very well and I wanted to do that in Ruby, and I knew Ruby ran well in OS10 and Windows, Linux and everything else. But the user interface was the issue, one of Ruby's weak areas. Rails was there but we didn't want to build an HTML interface we wanted something richer than that. We wanted the user experience to be richer, but not necessarily native to the OS. And the native frameworks for Ruby WX, widgets, they don't look very nice, they are not there yet, but we wanted a user interface that was truly cross platform. So what we chose as a technology was Flash, Macromedia Flash. Originally the idea was to use Macromedia's UI widgets and Ruby will basically drive that. About 2 years before that I'd written a bridge between Ruby and Flash that let me dot-method invoke objects in Flash from Ruby over socket
23. You write the exact Action Script in Ruby?
Exactly, you write the Action Script up here but then Ruby doing basically the system's behavior could drive that interface. You could send data up to the interface and pop a button up or pup a dialog up, whatever you wanted to do. We started doing that in the Flash widget framework, at the time that was Macromedia they got bought by Adobe subsequently but Macromedia's framework was not very robust. They since came along with Flex and it's a lot better, but at the time that was pretty bad. We wrote our own, and we needed to build a widget framework, it's not really what we wanted to do. So from the spring of 2005 until the fall of that, we built a user interface framework and we open sourced that, and that user interface framework is called Action Step, and it's a port of open step, the API, that's on OS 10, it's what the next step is which became Cocoa on OSX, it's in the application kit. We ported the entire application kit to Action Script2 it's about 60.000 lines of script. We open sourced that. You could actually take Objective C code that runs on OSX, and port it to JavaScript, well Action Script and it will run. And the GNUstep implementation, which is basically an open source implementation of open step, we actually build Action Step. And I have 2 other folks that helped me on this: Scott Hyndman up in Canada actually developed a huge amount of it and Ray Chuan from Taiwan worked on this as well. So the 3 of us basically built Action Step up. We were building something on top of it, so we wanted to build our whole application on top of it. So that was the Indie interface and then the Indie runs some underneath. We built this information environment application, it had a calendar, address book, we could synchronize with existing sources like your ... on your Mac, your address book there, your ... system and we could synchronize with Outlook, take the USB drive, plug it in sync to take your data down and then view it ... in this Flash based calendar, with your address book and then if you changed stuff it will synchronize back when you got to that computer. Or you could synchronize to as many computers as you want. It was a really nice system. We had to solve a lot of problems we'll be talking about tomorrow.
24. You speak in the past tense
Right, because in the spring of this year we released the beta of Indie, out to a community we had about 350 folks that were using Indie, and we got a lot of feedback on it. And the biggest feedback we got was "you have this really cool environment, out of browser experience, but it's a really rich graphical environment. Could you solve the my space problem?" We thought "what's the my space problem?" People today are sharing through my space, sharing to this public space, but they are really sharing with the people they know, but they are doing it in a public form and so there are these unattended audiences that come around these things and observe them. And that's where privacy issues happen and everything else. But the kids who share through social networks are just trying to share with each other, the people they know and to them what the my space problem was "could we do small groups rights?". We were talking to Clay Shirky and he said "small groups, we just don't do right" and so we took it at heart and said maybe we can solve that problem with Indi.
The next release of Indi, which is going to be in about a month, should be able to get from indi.com, basically we built on top of this portable environment which is still an information environment and everything else, is the ability for groups to be able to set up channels, and then share those channels, and you can share any kind of object. You can share mp3 recordings, like a broadcast you got, you can put it out there and you can share it with other people. You can share photos, but what's nice is we are running a native environment, so you can drag your photos out of I photos drop them on Indie and just share them. Now, everything is encrypted all the time in Indie, which means that as the stuff passes through the cloud and it does go through the cloud because some people are on some people are off, we want to support a synchronous communication. Going to the cloud is all encrypted. But only the people that you invite into those channels get the keys. So we don't know anything that is being shared, we have literally zero knowledge of what's being shared by architecture. Now the other problem is how do you scale storage system to be able to handle the transfer of all this stuff. Fortunately you didn't have to, because last spring Amazon released S3 so we have a Rails service which coordinates Indie's transfers of information, but the Rails service does not hold any data all that is on S3. The Rails service actually hand out your Rails to Indi to transfer things but everything is up on S3, everything is encrypted.
25. So S3 is real?
S3 is real and we are using it and it works great for what we are doing. Doesn't have to be real time, but it's always synchronous and works very well. This whole sharing channels thing which is built on top of Indi to allow people to privately share with groups, is also in the next release of Indi. And when you run Indi you double click on the executable on OS10 or Windows and it's Ruby underneath there which is running this entire thing and tat Flash interface is really driven by a whole Ruby core. And Ruby gives us an incredible amount of flexibility under the covers to do that.