OpenTerracotta’s DSO (distributed shared objects) allows the transparent clustering of user-defined classes, Java primitives, and even core Java class libraries, including its concurrency abstractions. DSO is a hub and spoke architecture, with a central server replicating data across JVMs as needed. Objects retain identity across a cluster, and the central data store (which itself can be replicable) ensures smooth failover. Terracotta replicates changes from one heap to heaps in other VM's. Terracotta also coordinates synchronization between threads, and can migrate threads across server instances. Bytecode instrumentation of classes declared clusterable is also used, resulting in fine-grained replication field-level changes across the cluster, without the need to declare objects as serializeable.
Terracotta for Spring is a customization of DSO which allows the transparent clustering of Spring Application Contexts, Singleton bean types, Application context events, as well as Spring’s JMX support.
Terracotta Sessions is also a customization of DSO that can cluster HTTPSession objects, supporting Apache Tomcat, BEA WebLogic and Websphere CE. Being based on DSO, Terracotta sessions was proven faster than other open source session clustering alternatives due to it's ability to distribute only field-level changes and also push changes only to servers that needed them.
Terracotta can also be used as a Grid solution, implementing the Master/Worker pattern defined in the CommonJ spec from IBM and BEA.
InfoQ spoke to Terracotta founder and CTO Ari Zilka to learn more about how and why this VC-backed startup went open source.
InfoQ asked how Ari was able to convince the investors to make such a drastic change to Terracotta's business model. Ari explained that Terracotta had board members supporting both sides, but that two datapoints easly aligned the board. Firstly, 80% of installs of Terracotta were in environments that were mixed with open source, usually Spring, Struts, and Tomcat. Second, many companies told Terracotta that the process of going through a proof of concept and evaluation is too time consuming from both a practical and legal standpoint, but if Terracotta were already out as open source then we "would have already been using you, running in production and we would have been calling you for support."
In terms of licensing, Terracotta created the Terracotta Public License which extends the Mozilla Public License with an attribution clause requiring companies shipping with Terracotta to include the equivalent of a 'powered by' logo. Ari explalined:
We wanted MPL, we needed a license that was a house hold name. We liked GPL from a business safety perspective but the community seems against it because of its viral nature... we wanted people to be able to adopt the software, both ISV and end user, minimize forking and get improvements back into the product, and attribution ensures that direct competitors have to give us credit which would give us more protection from forking.Comparing License contract revenues with support contract revenues, Ari explained that all the legalize associated with buying commercial licenses, idemnities and escrow issues is daunting, whereas with a subscription support model:
You pay a certain amount per year, and that amount is much less than a license so you don't have customers asking for a 90% discount... Subscription means that you can hold to your prices as long as they are commensurate with your costs... No one asks for source code escrow for example because it's open source. We don't need to define trigger provisions that give you rights to Terracotta's source code and I don't need to protect it because it's already open... Our support offering would make us more money after about 5 years than a licnense fee upfront but we have to earn the customers subscriptions year on year and provide value.On how he would position Terracotta to other open source clustering solutions, Ari explained that for small to medium companies, Terracotta gives you the developer benefits, the operator (datacenter) benefits and lower cost of ownership:
The core value proposition for Terracotta includes performance, ease of development, and ease of operations. Object identity works for clustering at heap level which means you're basically building to stateful model, which nobody else can say. For example, if you' were using serialization you would have to do things differently as a result of clustering than you would have done in a prototype of the same application. For ease of development thus I think we win. For ease of operations we are as good as anything else out there. We have a central server, which can be managed from one console, but peer to peer technologies can do that with JMX broadcast and such. And then our server clusters so there is no single point of failure so we're similar to other solutions. I would position it to an operator that you get the benefits of stateless while developers on your team get the benefits of stateful and we are the only solution that has both... And then from a scalability perspective I think we are one of the youngest companies out there and there are more scalable solutions from us, but we are scalable enough for small to medium companies.Moving onto more technical topics, Ari explained how Terracotta is typically being used:
Developers use Terracotta to share objects across JVMs (notice, we didn’t say "data" because sharing objects requires cluster-wide object identity and Terracotta’s customers rely on that). Specifically, developers write applications that contain more state in-Heap. If any one node fails, all other JVMs have access to the same data so nothing is lost; sort of a stateful model in a stateless runtime environment. Also, clustered apps scale well with our fine-grained replication (field-level) and fine-grained locking (object-level).Asked if people are actually clustering Java primitives and using standard Java concurrency features in clustered fashion, Ari gave some examples:
Clustered apps that were previously leveraging JMS and RMI to do things like scatter-gather / MapReduce / Master-worker and the like, are moving to a pure POJO model with LinkedBlockingQueues and wait() / notify() that clusters as part of the runtime. They switch because their current architecture is too intrusive or hard to maintain or not scaling as a function of nodes/size of clustered data.While Ari was careful to position Terracotta as a no-brainer answer for small to medium companies, he also explained how one customer was using Terracotta in a large-scale usecase:
Large can be defined as a large number of JVMs, large amounts of clustered data, or a significant rate of change to objects. Under one scenario, we encountered all three with thousands of JVMs and upwards of 80,000 transactions per second. The customer helped us design a solution where their application split objects across Terracotta servers by hashing the objects into separate hashmaps, each of which was accessible by all JVMs. In each map the app stored one-nth of the objects in the system. Then, we tuned Terracotta to not prefetch / batch objects and, furthermore, not to cache in the local JVM. There was no load balancer required because Locality of Reference was set to the Terracotta Servers. And, there were 10 of them. Each one was responsible for 8,000 transactions per second. This worked very well for the customer and outran many other options they had tried. While they had to write code to get the app to scale, the objects were still POJO and the customer was satisfied that they could continue to scale as their application grew.On potential misuses of Terracotta:
Terracotta is location-transparent meaning locality of reference is not maintained. State can be faulted-in from the Terracotta server as needed. However crossing JVMs implies that the calling thread must incur network overhead and latency costs. So it is recommended that applications maintain Locality of Reference as much as possible.On how Terracotta ensures fault tolerance considering its central coordinating hub, Ari explained:
In persistence mode, Terracotta allows state to be persisted across JVM restarts. However, it is not a system-of-record and lacks database features such as recovery, backup etc. So use it as a persistent data-store carefully only in situations where occasional data-loss in edge-cases might be allowable.
In our operational past w/ several large Internet sites, hub-and-spoke lent itself well to infrastructure services; operators know how to make machines fault tolerant and services highly available. Terracotta’s Server leverages this now-common-place Active/Hot-Standby architecture. Your application’s JVMs transparently connect to the standby Terracotta Server upon failure of the active Terracotta server. The design avoids split-brain / network partitioning issues. Honestly, we will have to start clustering Terracotta servers in a peer-to-peer fashion once a use case outstrips the capacity of a single active Terracotta instance. For now though, the fact that we push as much data as changes in your objects means that even the largest installations have not outgrown a single server instance.Finally, on how Terracotta supports data partitioning architectures (co-locating data access with data residency to avoid the latency of inter-node data-sharing), Ari explained:
Terracotta has implemented the Master/Worker pattern.It is intended to help with Locality of Reference by moving the app to the data instead of the data to the app, but it requires uses POJOs to function. Our customers call this many things from “grid” to “Master / Worker” to MapReduce to scatter / gather. Conceptually, work gets input to a queue and is processed by multiple, collaborating worker-JVMs, each of which owns some subset of the data. The pattern provides all the required plumbing to route data to workers and keep it there (off of other workers and off the network).An Enterprise edition will also be available for purchase, but it is identical to open, except that there is a contract with indemnification and perpetual license to a particular version of TC's product.
In terms of community integration with Terracotta, Ari told InfoQ that Tomcat support company Covalent will be bundling Terracotta in their Tomcat distribution. The Geronimo team is considering bundling Terracotta in the Websphere CE as a clustering option. Terracotta will be linked to as clustering option on the Tomcat clustering FAQ and by Glassfish. Apache Tomcat and Tribes committer Filip Hanik is also a committer on openTerracotta, as is Jeff Genender, Geronimo committer.