I'm a developer; I work at Songkick.com in London. We're a company specializing in upcoming gig listings. I'm also a Ruby enthusiast, that's why I'm here at this conference - that's me.
As I said, we're about upcoming gig listings, so we have over 110,000 upcoming gigs listed in our database, we pull from multiple ticket vendors, so if we don't have an upcoming gig in our database that you want to go to, that's a bug for us. We have over 1.4 million events stretching back to the '60s, so if you have a shoe box full of gig tickets, that you want to share with the world, you can do that on Songkick. So, that's Songkick.
3. How do you get your data? How do you get your ticket data?
We have partnerships with a lot of ticket vendors. They provide us with feeds of their data, which we import and translate into nice listings that you can view on Songkick.com. If you are looking for a gig, we'll often offer you many choices of tickets that you can purchase on different vendors with prices attached. You can make a choice upfront about where you want to get your tickets from and click through to the vendor ticket from there.
4. What technology do you use to build Songkick? Ruby?
We're exclusively Ruby at Songkick. I think that is still true. We use Rails for our websites, several different versions and of course we use many hundreds of different gems. We use Ruby extensively, so everyone at Songkick is a Ruby developer and we use Ruby for the website, for our API, we use Ruby on the backend for our daemons to do various kind of job processing, for scraping we have libraries written in Ruby for pulling in data from all the sites. All of our system administration is done using Ruby. We're exclusively a Rubyshop and we're finding it's working really well for us.
5. What Ruby libraries do you use to pull the data from other providers?
We have custom written stuff for that. We have a scraper library that manages sets of scrapers that run against different ticket vendor sites, so we can monitor that through our interface and check that they are all green.
6. Which Ruby do you use currently? 1.8, 1.9?
We're on MRI 1.8.7 right now. Actually, we're using Ruby Enterprise Edition, which we found to be really worthwhile switching to. We have not got any plans to move to 1.9 just yet, but it's becoming more imaginable that we do that in the near future.
Myself and my colleague Matt Wynne, gave a talk at this conference. It was about using NoSQL databases with Rails, but it was also about SQL databases. Our talk was on how to move from one side to the other across that divide because there are plenty of people here who would like to use NoSQL databases, if they get the chance, but most of us are making our money maintaining perhaps large applications running on top of MySQL. It's not straightforward for us to just start again from scratch or begin a new project that uses it. How can you really bring in a document database into an existing relational solution in a principled and pragmatic way? That was the subject of the talk.
I think that using NoSQL offers a lot of advantages. I want edto make it clear that it's possible to do this without having to start again from scratch. At Songkick we developed a few libraries in the process of figuring how to do this and our talk was really about presenting those libraries and showing how they work together to make this possible. One of those was a replacement for the Rails observers, which are built into Rails, which allow you to do asynchronous observers, which run out of process, but at the same time work with Rails' existing way of doing things. It's a very understandable solution for someone who's used to Rails to start using asynchronous observers. We haven't released that yet, but we'd like to release that as open source at some point. We also implemented a library for memoizing our data into MongoDB and MongoDB is a new document-oriented database that we've had a good use of, as well as libraries.
Our talk was also about the pattern of introducing presenters into our Rails application, which is something that it's been debated back and forth inside the Rails community. We've always been a little unsure as to where they fit, but with this new goal of trying to denormalize our database and bring together all of the different pieces of data that are needed for our website and our Rails project into one place, the use of presenters was really useful for us because we use them as a second persistence layer on top of our models, in order to give the website exactly what it needs, when it needs it. We think there is more room to talk about presenters in the Rails community and we're going to try and put some blog posts up that explain what we've been doing.
8. You say the asynchronous observers run out of process, is that right?
They do. There were 2 ways we could have done this. The first was to implement a kind of an ad hoc messaging system or message format for every job we want to do in our system and that would work really well. We started going in that direction, but what we found with the asynchronous observers was we’ve implemented a system where every ActiveRecord event (so a creation of a database row, an update of a database row or a deletion) gets serialized into a message representing that operation and puts it on to the queue. We do that for everything that happens pretty much. What we've got is we call it a megafeed, it's like a firehose of events, thousands of them coming all the time.
We can listen to that and dispatch on the events in our system in our job processing things without having to worry about changing the website to support a new kind of message or a new kind of job. We often find ourselves spinning up new demons to deal with, say image resizing to generate thumbnails of different sizes for the website. But we don't need to go and change our website in any way because the megafeed of active record events is already enough information to handle that. Or say we want to send an email to a new user, we just listen to the megafeed and if the new user event comes down the queue, we pick it off, we get the information out of it and we send the email.
It's a really good abstraction because it unifies all the different kinds of thing we want to do into one type of message and we really have found ourselves going beyond that because everything we want to do is embodied in some ActiveRecord operation somewhere.
9. So it's a kind of event bus?
Yes, we use RabbitMQ and we send all the messages of RabbitMQ and Rabbit has been a great solution for us. We found ourselves using it more and more all the time. They are just around the corner from us in London as well, we must meet up.
10. You mentioned MongoDB. What's so great about MongoDB?
It's pretty quick, for one thing, which is important when you are doing this essentially as a caching layer. It's also really flexible and since we didn't know exactly which direction we would take in this project, we found that really useful. With Mongo you can store documents, you can use it as key/value store, you can store documents, you can index around the contents of those documents, perform complex queries against them, which means that we didn't have to precisely define what we were going to do upfront and that's been really valuable.
We're using Mongo as a key value store for documents, but with the addition of being able to query over them, we're using it to store HTML fragments and we're using it to store indexes of things like user IDs in lists and Mongo supports things like atomic operations on lists so that was really useful to us as well. I think the real benefit of Mongo is it allows you to stay flexible when you are not entirely sure of which direction you're going. But there are lots of good alternatives in the document database sphere, so we wouldn't be adverse to trying a different solution if one came along.
11. How did you choose MongoDB or why did you choose MongoDB over others?
Like I said, there were lots of good options. I would be interested in using Redis more, but at this point it's really hard to see if any of them are really so much better than the others. In a sense, we used Mongo internally for a while because we all try out new databases and it worked for us on an admin application and the fact that it was already working for us is essentially a big reason why we chose it. The fact that we can stay flexible with it is another big reason. We're probably in the long run still undecided about which database we'll go with, but Mongo's working for us for now, working great, so that's enough for us for now.
Yes, I had a lot of fun writing that. It's still at a very early stage and there are things that both need to be improved with it and things that are broken and need to be fixed. I think with JRuby, there is a great opportunity to do some excellent profiling solutions if you want to, so I'm looking forward to getting into it more and releasing a new version soon.
13. How is JRuby Prof better than a JVM profiler?
Before I wrote JRuby Prof, which I needed for one of my projects, I looked around what the other options are. There is a pure Ruby profiler that comes with any Ruby and that works, but it's not really good enough for the production use. After that I took a look around at the Java world’s profiling stuff and I'm not someone who knows a lot about Java so it was all new to me. I looked at Visual VM, which is Sun's product and that worked pretty well, but it did segfault my JVM on occasion I was trying to do some actual profiling, so I kind of dropped that fairly quickly. Then Charles Nutter recommended YourKit to me, which worked a lot better and I liked that a lot, that's a commercial product.
I think there are a lot of commercial products out there like JXInsight that I'd love to have a go at using but they are just a bit too expensive for me, for what is essentially a hobby project. Between those 2 pure Ruby and the full on Java Profilers I was looking for something that was very much like the tools we use in the Ruby community, which didn't actually exist for JRuby at that time. It was fairly simple to get something up and running straight away, which is why I say this is a new project that needs work, but it does what you need it to do. It's fairly low overhead, it profiles your method tree and builds a call graph, so you can see exactly which parts of your code are taking the time, which is exactly what I needed and hope that you will find it useful as well.
14. What technologies do you use to implement JRuby Prof? How do you hook into the Runtime?
JRuby supports Ruby events, which are very similar to MRIs, presumably they've been in there since the beginning of the project. It was pretty easy to write a Java library that just hooks into those events and listens to. The events cover method calls, method returns, exceptions, things like that that happen in the calls in your code. Hooking into those gives you enough information to start building a call tree of methods and durations.
15. Your other JRuby project is something called RedCar. What is that?
It's a text editor. I've been writing it for a little while now. It's currently implemented in JRuby, it has been working on JRuby since the start of the year. It's implemented in Ruby pretty much from the top to the bottom. The only thing that's pushed down to Java is syntax highlighting which needs to be really quick. The fact that it's written in Ruby, I think for Rubyists make it a great editor to start using because at any time you can just pull up a tab and start typing in Ruby code, which it will execute inside itself. You can implement a new Ruby command for RedCar, say if you want to do something with your text in a buffer or something, you can implement a command straight in there and have it pop up in the menus immediately.
You don't even need to restart your editor. Using Ruby's dynamic nature to support editing the editor from inside itself is really good fun. In general, I've tried to make it a good solution for Rubyists because we like to be able to fiddle with everything right down to the ground. Everything in RedCar is implemented as a plug-in and all the plug-ins use the same APIs as anything else. There is no hidden core in the editor beyond which you can't hack. If you open up the code, if you open up a REPL inside RedCar you can start changing any portion of it.
That causes some problems because in Ruby you can't stop anyone doing anything. We have to be careful about making sure that our APIs are really good enough to support what people want to do, but I'm really excited about projects and I hope that this conference would get some more users.
16. What GUI library do you use to build RedCar?
We use SWT, which is the same GUI library that was developed for the Eclipse project and it's terrific, it uses native widgets on all 3 platforms. Actually I believe it still works on Motif as well, if you fancy that. It's been great. It works really well and we haven't had any problems with that tool.
17. Do you use just basic SWT, or do you use any of the OSGI libraries, or anything like that?
We use basic SWT in the main, but we also use some of the JFace classes on top of that which provide some convenience things on top of that, which are useful because things like the main one we use the undo manager from Eclipse, which we haven't. I would hope to replace with the pure Ruby version soon enough. We did that with the clipboard. Initially we used the JFace clipboard, but we switched to pure Ruby version when we found the time.
18. You have some integration with TextMate in RedCar, is that true?
Yes. TextMate's a terrific editor and in fact I started RedCar probably mostly because I was on Linux and wanted something that worked like TextMate. I don't know how familiar you are with TextMate, but most of its themes and syntax definitions are included in a set of open source bundles that get linked in. That's been great for TextMate because it's meant that anyone can edit those and improve the editor in that way, which is great.
The author of the TextMate, Allan Odgaard is good enough to be supportive of projects that make use of those bundles and without them, RedCar probably wouldn't exist. So a big "Thank you!" to Allan for making those open source, which has been terrific for us. RedCar supports the same themes and snippets and syntax definitions as TextMate.
19. Is RedCar an editor for single Ruby files or do you have the concept of a project?
We have a limited concept of a project. Red Car is still pretty new, so there is a long list of things we need to implement, like I said, hoping that contributors will come forward at this conference. The project's of course really simple at the moment, but it's enough for me to use RedCar full time in my job and at home, which is to me the important thing. We're starting to get more project support, so someone has come forward online with GitHub project browser, which we enjoyed and someone came forward with the ctags integration for RedCar, which I can't wait to get working with. We can have ctags support in RedCar as well.
20. That's for code browsing, essentially?
Ctags is something that I think it's more common for Emacs users, but it's a way to jump to the definition of a method or a class from where it's being used. It's used everywhere and it's good cross platforms.
I'm not a fan of big complicated dialogs in my editors or IDEs. One thing we're trying to use in place of that in RedCar is HTML views, HTML tabs. A big part of the draw of RedCar is you can write extensions for it in the language that you already know. Having to learn a GUI API, with even something as good as SWT is not a big draw for most people. I would like the ability to write plug-ins for Red Car that use skills I already have, like HTML, CSS and JavaScript and that's what we're trying to do with these web tabs. How they allow you to use JavaScript which then the RedCar runtime calls back into Ruby and that's really transparent.
You write a controller with some Ruby methods on it and Red Car will put that into the JavaScript and let you call those methods from the JavaScript. This is going to allow us to hopefully replace most of the complex dialogs with HTML forms, with JavaScript. It will work a bit like Ajax in that it will be instantaneous, you won't have to reload the page or anything like that. I'm hoping that people will be able to use their existing skills to write plug-ins for Red Car like this.
22. You are using the SWT HTML component and you expose the Ruby APIs and the Java APIs?
Right. We couldn't do without the SWT support, which is terrific. SWT allows you to bind JavaScript methods to Java methods and then JRuby has such good tight integration with Java that is very simple to then have the JavaScript be binding to the Java, which binds to the JRuby. It all chains together and I was actually impressed with how well these libraries all worked together to make that possible. It was a bit of a surprise to me but is very welcome.
The plug-in authors can use whatever they like to. As yet I'm undecided as to how far the helpers will go as to bring in default things, but at the moment you can use whatever you want.
24. Why do you suggest using these HTML views over building your own little SWT window or widget?
Essentially because I personally find them nicer to work with. People already have skills in HTML and JavaScript and CSS and they can bring them to bear immediately on improving their editor and writing new commands and new plug-ins. Having to learn a GUI system is not a big barrier and for more complex things or for tighter integration it will still be required, but there is a huge class of things that you do in a standard IDE, like the search results for instance, which could be presented as HTML.
The preferences pane or the plug-in manager or the debugger perhaps - there are many things that can go into this. It's really about making it easy to extend by its users, it's the main reason.
25. Where could the audience go and find RedCar and JRuby Prof?
JRuby Prof is available as gem from GemCutter. "sudo gem install jrubyprof", will do it. You need to install it as a JRuby gem. RedCar is also available as a gem and even though it runs on JRuby you can install it, you just need to have any Ruby or Ruby gems installed and it will go and fetch all the JRuby jars that it needs to bootstrap itself. "sudo gem install redcar" will do it.