Airbnb recently opensourced Airflow, its own data workflow management framework. Airflow is being used internally at Airbnb to build, monitor and adjust data pipelines. Airflow’s creator, Maxime Beauchemin and Agari’s Data Architect and one of the framework’s early adopters Siddharth Anand discuss about Airflow, where it can be of use and future plans.
Snowflake Computing has announced the general availability of their Snowflake Elastic Data Warehouse, a software as a service offering that provides a SQL data warehouse on top of Amazon Web Services.
Metanautix recently announced the newest version of its product, Quest. Quest allows enterprises to build software defined data marts that can run in virtualized servers. Designed from the ground up with security and auditability in mind, Quest can deal with Big Data workloads and encapsulate it into different logical views, ready for consumption by different departments in enterprise.
This post talks about using an agile implementation for data warehouse projects.
Google has unveiled their new data-warehouse called Mesa. Mesa is a system that scales across multiple data centers and processes petabytes of data, while being able to respond to queries in sub-second time and maintain ACID properties.
Teradata revamps its cloud offering, offers Data Warehouse Data Platform as a Service solution. Teradata Cloud is aspiring to become a worthy competitor to Amazon Redshift, with a richer set of predefined libraries and a more effective way of loading data.
The AWS team has announced a limited preview of Amazon Redshift, a cloud-hosted data warehouse whose cost and capabilities are poised to disrupt the industry. In addition, AWS revealed two new massive compute instance types, and a data integration tool called Data Pipeline.
Eobject.org's open-source Java framework MetaModel implements a unified API for the access, exploration, and query of different datastores. Eobjects.org, both a website and an open source software organization dedicated to "the development of Open Source software related to Business Intelligence and Data Warehousing", has recently published version 1.5 of MetaModel.
The Hadoop Summit of 2010 included presentations from a number of large scale users of Hadoop and related technologies. Notably, Facebook presented a keynote and details information about their use of Hive for analytics. Mike Schroepfer, Facebook's VP of Engineering delivered a keynote describing the scale of their data processing with Hadoop.
The need for machine-learning techniques like clustering, collaborative filtering, and categorization has steadily increased the last decade along with the number of solutions needing quick and efficient algorithms to transform vast amounts of raw data into relevant information. Apache Mount 0.3 has been announced on March, adding more functionality, stability and performance.
Dan Pritchett suggests that analyzing streams of events using Event Stream Processor could be an interesting alternative solution to data warehousing applications, which have, in his opinion, important downsides in terms of cost, scalability and reactivity.
Large centrally designed BI systems often don't meet the expectations of their end users. In this article at Cutter IT journal Scott Ambler has written about using Agile methods to help meet the user's expectations and deliver business value quickly.
Michael Stonebraker, co-founder of the Ingres and Postgres relational database management systems (RDBMS) and CTO of Vertica Systems, laid the framework for a debate in the database community by declaring that most major databases should be considered legacy technology.
ActiveWarehouse, is a significant new plugin that makes it easier to build data warehouses in Rails.
Some people think they can only be Agile with small, co-located teams and full management support, but most teams aren't that lucky. So, should they should give up on Agile techniques? Scott Ambler's answer is a resounding "No!" His Dr. Dobbs article "Imperfectly Agile: You Too Can Be Agile!" outlines how Agilists overcome common challenges that others use as excuses for not being Agile.