In-Memory Databases Reaching Tipping Point

by Roopesh Shenoy on Apr 26, 2012 |

In-Memory Databases are very near to becoming widely adopted, says David Campbell of Microsoft in his article “The coming in-memory database tipping point”. He also explains Microsoft’s strategy around this space.

According to David, following trends will combine to make In-memory databases common in 5 years -

Bandwidth and capacity (Hard disks, RAM) tend to increase much faster than reduction in latency
Multi-Core CPUs that need to be used effectively
Ability to randomly access small amounts of data on disk not improving in pace with sequential bandwidth of reading from disk

Leading to new design approaches by database designers -

Compression to hold more data in RAM – decompressing data when needed tends to take lesser CPU cycles than reading it off the disk, by several orders of magnitude
New Database algorithms – for e.g. Column based approaches v.s. the traditional row based approaches, to spread load over multi-core systems

Gartner has identified in-memory computing as one of the emerging trends in their research paper, “The Top 10 Technology Trends for 2012”. Other vendors too are already positioning themselves to cater to this market – SAP for instance with its’ HANA platform, and Oracle with it’s TimesTen product.

What is Microsoft doing to ride this trend? David explains -

We have created a column based storage engine which ships as part of the “PowerPivot” add-in for Microsoft Excel. In SQL Server 2012, this ships as the xVelocity in-memory analytics engine as part of SQL Server Analysis Services.

Our technical approach considers not only best-in-class capability but how to integrate and deliver as a part of a complete data platform – ultimately the highest value for customers will be achieved this way.

In-memory databases are not new in terms of technology – we already have several popular persistent and transient databases such as Redis and Memcached being used for caching or low-latency data-access. However even traditional relational database providers catering to Enterprise customers are now taking in-memory computing seriously, which is interesting to observe.

Related Editorial

Betfair Chooses not to Gamble with Database Performance: An AppDyanmics Case Study

Impact of Java EE Frameworks on the Structural Quality of Applications

Why NoSQL - Three Trends Disrupting the Database Status Quo

Making the Shift from Relational to NoSQL

Building Content-Driven Apps - Couchbase and Elasticsearch for Content and Metadata Stores

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Community comments

Is This Really News? by Chris Alexander Posted

Re: Is This Really News? by Roopesh Shenoy Posted

Re: Is This Really News? by Cameron Purdy Posted

Re: Is This Really News? by peter lin Posted

Re: Is This Really News? by Roopesh Shenoy Posted

Re: Is This Really News? by David Ventimiglia Posted

maybe a few beers were tipped before microsoft did the press releases by John S Wolter Posted

Is This Really News? by Chris Alexander

I am perplexed why InfoQ titled this article "In-Memory Databases Reaching Tipping Point." For many of us DEVs working on low latency applications, in-memory databases have been a standard component of our architecture. I think the real question is: is this now the case because MS is publicly admitting they're coming to the "in-memory" game, late yet again. There's no tipping here! Since 99% of the news-worthy, cutting edge applications and platforms rolled-out over the past 7 years do not rely on M$ tools, the article should be entitled, "MS Brings In-Memory Databases to Laggard, Slow-To-Adopt Enterprises." This is a tipping point for enterprise IS departments who are hook-line and sinker MS shops, not businesses/shops with entrepreneurial-like IT leadership. Personally, looks like this article was written on an iPhone after the author touched-based with former MS contacts. Definitely no novel reporting here. You can do better InfoQ!

Re: Is This Really News? by Roopesh Shenoy

Chris, thanks for the feedback.

Like you point out, the technology is not new at all. There is also no doubt that Microsoft lags others, even the likes of Oracle and SAP who themselves lag others in this field. The tipping point here however refers to a much more wide-spread adoption - this includes the slow-to-adopt enterprises (and not just MS-shops!) which are undoubtedly much larger in number.

Also the application of in-memory today is focussed primarily on caching or low-latency needs, often supplemented with a more traditional store. I would argue that it is still unusual to consider your in-memory database as a primary datastore. That has potential to change, with the hard-disk being used just like you would use back-up tape today.

maybe a few beers were tipped before microsoft did the press releases by John S Wolter

This is not new. Flash drives and other quick access methods have always been out there. The press release used all the right words and phrases. Examples include "tipping point", "leveraged", "disruptive", "breathtaking"....

Okay everyone get back to work.....

Re: Is This Really News? by Cameron Purdy

.. even the likes of Oracle [..] who themselves lag others in this field ..

Well, that was random!

You mentioned as two examples "Compression to hold more data in RAM" and "Column based approaches", both of which have been implemented in the Oracle database now for years (see Exadata for example) -- not to mention in TimesTen for well over a decade.

Oracle today has the most successful in-memory database product in the market with TimesTen, and the most successful in-memory data grid with Coherence.

Please don't make up for one substantive gap by re-directing and creating another.

Peace,

Cameron Purdy | Oracle

Re: Is This Really News? by peter lin

I was tempted to make the same point, but I figure others would notice the error.

Maybe Roopesh meant Oracle had to buy those products. Even then, Oracle bought TimesTen and Coherence a while back, so "lagging" is inaccurate.

Re: Is This Really News? by Roopesh Shenoy

I need to clarify - I din't mean lagging currently, meant lagging in terms of time of entry into the field. Apologies if I seemed to mean otherwise.

Re: Is This Really News? by David Ventimiglia

Well in fairness, the author wrote right at the top of the article what he meant by "tipping point": becoming widely adopted. While people like you who demand low latency may have been using in memory databases, that does not necessarily mean the technology has been widely adopted.