ActiveWarehouse, a New Step for Enterprise Ruby

Posted by Sebastien Auvray on Mar 28, 2007 12:00 PM

Community

Ruby

Topics

Data Warehousing ,

Ruby on Rails

Tags

ETL ,

Rails Plugins ,

Data Warehouse

Ruby and Rails continue to generate a lot of buzz in the overall software development community, but debate rages over how it will accomodate enterprise needs. For instance, is Rails able to handle large amounts of business data? With the new release of ActiveWarehouse, open-source Rails programmer Anthony Eden delivers a plugin that makes it easier to effectively build data warehouses using Ruby on Rails. Right now, ActiveWarehouse is one of the most active RubyForge projects, and progress along the features roadmap looks impressive.

The ActiveWarehouse plugin simplifies building data warehouses in Rails. A data warehouse is a database designed specifically for analytical use as opposed to operational transaction processing. Typically a data warehouse houses data which spans several years and is sourced from numerous operational databases. Data warehouses are usually highly de-normalized which is contrary to transactional systems which tend to be normalized. The links in the side bar provide additional information.

In this release you'll find notably : generators for facts, dimensions and cubes; Multi-dimension support; Automatic creation of aggregates and many other features, with a full pipeline of planned enhancements on the way.

To get the data from multiple data sources into the data warehouse, ActiveWarehouse is paired with the ActiveWarehouse ETL component.

The ETL handles most of the basic source types which would be used when integrating a fairly current system (delimited, fixed-width, XML and database sources). It can also be extended with custom parsers, so that part is handled at the moment. There are also enough transforms to be useful and adding new ones is easy. The system is definitely extensible.

Other functionalities also available are notably : Virtual source fields; Support for pre and post processing code; ETL Domain Specific Language (DSL) control files. Bulk loading is available only for MySQL at the moment. Anthony is also working on performance issues, which are always a crucial in this domain.

ActiveWarehouse and ETL component abilities are demonstrated through a comprehensive tutorial.

1. Back to top

   Looks Good

   Mar 28, 2007 1:27 PM by Frank Bolander

   Good job Anthony. The ETL stuff looks very interesting. Any plans on supporting MDX?

   Reply

2. Back to top

   Re: Looks Good

   Mar 29, 2007 1:32 PM by Anthony Eden

   No plans for MDX at the moment. ActiveWarehouse isn't a middleware app at the moment, rather it is designed to be used directly from Rails. That's not to say that it couldn't become a standalone middleware app in the future, it's just not something I or the other authors need at the moment. Feel free to jump in and scratch any itches that you have, though. :-)

   Reply

3. Back to top

   Enterprise

   Mar 30, 2007 9:50 AM by Steven Devijver

   If data warehouse code is considered "enterprise" then many of the scripts I wrote in my Perl days are "enterprise" too. Whatever your connotation of "enterprise" is, what it boils down to is how much your VM can take under load. The JVM can, Erlang too and there are others. The Ruby VM however is lagging many years and millions of dollar/euro behind. Don't take my word for it, ask the people that work on it.

   Reply

4. Back to top

   Re: Enterprise

   Jun 16, 2007 7:29 AM by Anthony Eden

   Steven, You are right, the performance of the Ruby interpreter *is* an issue in certain parts of ActiveWarehouse, specifically data aggregation for large data sets. I remember the same was true back in the early days of Java as well, so I have confidence that the performance of Ruby will continue to improve. One day we will have both performance and joy of development at the same time. ;-)

   Reply

• Older >