Java In-Memory Persistence with Space4J

Posted by Dionysios G. Synodinos on Sep 20, 2008

Sections

Architecture & Design,

Development,

Operations & Infrastructure

Topics

Java ,

Data Access ,

Performance & Scalability

Space4J is a simple database system that will let you work with Java Collections in memory. Since memory is several orders of magnitude faster than disk for random access to data, Space4J provides better scalability for "real-time" web applications and systems that require performance.

With Space4J Instead of having to perform a SQL SELECT to fetch a User from a database table, the developer accesses the users map (java.util.Map) and calls users.get(id). Since all data is kept in memory inside the JVM, there is no need for an extra database application, a socket connection, a JDBC driver, a SQL statement or any kind of ORM tool. Data is just there, inside objects, inside Java maps. For operations that modify data a Command object is created and then serialized and saved to disk in a log file. At restart the past commands are read from the log files and re-applied, generating the exact same data set you had for example before a system crash.

In order to prevent log files from becoming huge from time to time the application can take a snapshot of all data to disk. Space4J keeps all data inside the Space object. When taking a snapshot, the whole Space object is serialized and saved to disk. Therefore at restart only the commands since the last snapshot need to be re-applied, not all of them.  The size of the snapshot will depend on the application. Also the system has to enter read-only mode when saving the snapshot to disk, unless a Space4J cluster is used. An example of such a deployment would be a web application in load balance, where every web server would have its own Space4J node from a cluster

Space4J comes with a complete indexing framework that supports four different types of indexes to search data in a variety of ways. Also it can be used alongside a regular database  for offline work, data  warehousing, reports, etc.

Space4J uses the Java 1.6 concurrent data structures for concurrent read/write access to data so writers only block writers, readers don't block or get blocked by anything. This means that modifications are done one at a time while read-access operations are done concurrently without any ConcurrentModification exceptions!

 You can download the latest release (0.9.1) or browse through the source repository.

For a more information regarding the emerging paradigm of shifting data access from disk to memory for performance and other data access issues, you can read “RAM is the new disk...” by Steven Robbins, here at InfoQ.

Dionysios G. Synodinos is a Web Engineer and a freelance consultant, focusing on Web technologies

1. Back to top

   Prevayler

   by Peter Monks

   How does Space4J compare to Prevayler?

   Reply

2. Back to top

   Re: Prevayler

   by Ladislav Thon

   I wanted to ask very same question, but it is apparently answered on the product site:
   
   
   
   How Space4J compares to Prevalyer? Space4J and Prevayler are two free Java implementations of the same concept. Prevayler has the merit of being the first implementation created from scratch by Klaus Wuestefeld. Klaus also has the merit of pushing the idea of a prevalent system that is both possible and desirable in many cases. Although Space4J and Prevalyer are centered in the same idea, they have totally different APIs and implementations. It is pretty much like Struts and JSF for web frameworks.

   Reply

3. Back to top

   Broken?

   by Martin Probst

   I wonder if this is really worth the hassle. You will get into terrible problems once your data exceeds some GB, concurrency is difficult to get right for access over several collections (no transactions), and there is no rollback or similar.
   
   I think it's questionable if for such a limited approach if it's not easier and more straight forward to write your own simple write-ahead logging persistence, in particular if you have high performance requirements. And in other cases, just use a database.

   Reply

4. Back to top

   My 2 cents

   by Thai Dang Vu

   If you don't use SQL statements then it'll be difficult to switch to real database when we need more space than the memory can provide.

   Reply

5. Back to top

   Caching

   by Gleb Liferenko

   Such a technology should be used in combination with, not instead of a relational database, similar to how [[cached]] is used. Also, it would solve your transaction problems.

   Reply

6. Back to top

   Re: Broken?

   by Sergio Oliveira

   Martin said: concurrency is difficult to get right for access over several collections (no transactions)
   
   
   
   This difficult task was done for me by Doug Lee in the java.util.concurrent package. Like it is said in the article: "Space4J uses the Java 1.6 concurrent data structures for concurrent read/write access to data so writers only block writers, readers don't block or get blocked by anything. This means that modifications are done one at a time while read-access operations are done concurrently without any ConcurrentModification exceptions!"
   
   
   
   Martin said: there is no rollback or similar
   
   
   
   This is a drawback I agree. But commands should be atomic by nature and you can always check for errors before modifying. That's possible because every command is executed in isolation from the other ones, in other words, writes are serialized and never executed concurrently.
   
   
   
   Gleb said: Such a technology should be used in combination with, not instead of a relational database, similar to how [[cached]] is used. Also, it would solve your transaction problems.
   
   Yes, you got the correct idea!

   Reply

7. Back to top

   GC

   by Dax Abraham

   What about garbage collection when the size of the object in memory becomes really huge ...

   Reply

8. Back to top

   Re: GC

   by Sergio Oliveira

   You must be talking about the Space object that holds all other collections. This guy is never GC. There is no need to.
   
   You should think as Space4J as a relational database, but:
   
   Database = Space object
   Table = Java collection
   Index = Java Map
   Selects = direct collection access like map.get(id)
   Inserts/Updates/Deleted = Space4J command

   Reply

9. Back to top

   How is this different?

   by Sarath Chandra P

   From Object Caching?
   From memcached?

   Reply

10. Back to top

    Re: How is this different?

    by Sarath Chandra P

    oops.. I lost context! ignore my earlier question

    Reply

• Older >