Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Cassandra: The Definitive Guide, 2nd Edition Book Review and Interview

Cassandra: The Definitive Guide, 2nd Edition Book Review and Interview

Key takeaways

  • Learn about Cassandra NoSQL database version 3.0 features and capabilities
  • How to install and configure Cassandra database including the cluster management
  • Data modeling (conceptual, logical, and physical) with Cassandra database
  • Data partitioning and replication strategies offered by Cassandra
  • How to implement security for authentication, authorization, and encryption

Cassandra: The Definitive Guide, 2nd Edition book authored by Jeff Carpenter and Eben Hewitt covers the Cassandra NoSQL database version 3.0.

Authors discuss several different important topics related to this popular database, including data modeling (conceptual, logical and physical models) and Cassandra architecture detailing the consistency levels, partitioners, and replication strategies. High availability of Cassandra database cluster across multiple data centers is also covered. Readers will learn how to install and configure Cassandra including how to create a Cassandra Cluster. Authors provided dedicated chapters on monitoring and maintenance topics which are critical after Cassandra database applications are deployed in production environments.

InfoQ spoke with Jeff Carpenter about the second edition of the book and Cassandra database current features and future roadmap.

InfoQ: Why did you decide to write this book? It had been almost 6 years since the first edition.

Jeff Carpenter: The first edition by Eben Hewitt in 2010 was the first book about Cassandra. Because it was written against the 0.7 release, later developments like the Cassandra Query Language (CQL) were not covered at all. The book was still well regarded for how well it explained the core principles and architecture, but it was showing its age with references to obsolete terminology (column families, super column families, etc.) and implementation details (for example, the Thrift interface). Since the fundamentals of the book were really solid, my task was essentially to preserve what made the first edition special while bringing the particulars up to date.

InfoQ: There is some confusion on which NoSQL database category Cassandra better fits in, Key Value, Column Store, or Tabular. Can you please define what type of database is Cassandra?

Carpenter: This is a question which resurfaces periodically in the Apache Cassandra community.  Over time, the community has really coalesced around the term "partitioned row store". This means that Cassandra stores data in rows, which are mapped to partitions. The partitions are allocated across multiple nodes according to a replication strategy, which determines which nodes will store data with partition keys that map to a specific token range. The storage of each row is sparse, meaning that only “cells” containing data values will be stored.

InfoQ: What are the interesting new features in Cassandra 3.0 release compared to previous versions?

Carpenter: The 3.0 series represents a big leap forward for Cassandra. The major new features include materialized views and a new index implementation known as the Secondary Attached Storage Index (SASI). Materialized views in particular are very exciting, as they allow Cassandra nodes to take on the work of maintaining denormalized views, which have typically fallen on client applications.

Historically, users of Cassandra have advocated a query-first approach to data modeling. In this approach, you first model all anticipated queries and then create a table for each query. For example, I might create a table that stores hotel reservations by confirmation number. Let’s say that later on, I decide that I want to query the reservations by hotel and date. Prior to 3.0, I would have had to create a separate table with a different partition key to support this query. This causes me quite a bit of work: I have a data migration task to populate the new table, and I have to modify my reservation application to keep the tables in sync. On top of this, I’ve basically doubled my storage requirements for reservations.

With Cassandra 3.0, I can now create a materialized view on my “reservations_by_confirmation_number” table to support my new access pattern. Cassandra takes on the task of keeping the view in sync with the underlying table, and does it more efficiently than my application ever could.

There were some key changes under the hood that enabled the implementation of materialized views.  Cassandra’s underlying storage engine and SSTable file format were completely reworked to be more closely aligned with how data is represented CQL, and this yields many benefits in terms of improved read/write performance and reduced storage footprint.

I’m also really pleased with the SASI indexes. Cassandra has supported secondary indexes since the 0.7 release, but their use comes with a performance caveat since queries over these indexes typically require interacting with every node in the cluster in order to locate items referenced by the index. SASI indexes are a new implementation that performs much better than the original secondary index implementation, as an index is co-located with each SSTable file, rather than being stored as a separate “hidden” table as in the original implementation. More importantly, SASI indexes support LIKE semantics, which allow us to create queries with partial text matching. To get full-featured search you’ll still need to integrate with an engine such as SOLR, but this is a big step forward.

InfoQ: Can you discuss the partitioning and replication strategies offered by Cassandra and how they are different compared to other NoSQL databases?

Carpenter: The partitioning and replication strategies you select determine which nodes will store your data, so it’s important to be aware of the options and significance of each.

A partitioner is really just a hash function over the elements of a partition key, which produces a value called a token. The range of possible token values is referred to as a ring, and Cassandra allocates ranges of possible token values in the ring to the nodes in the cluster. So the partitioner generates the token value which is used to identify the primary node responsible for storing the data. The default partitioner is the Murmur3Partitioner, which is a random partitioner. There are also ordered partitioners. The goal of an ordered partitioner is to maximize the co-location of partitions with similar keys on the same node, in order to improve the performance of range queries. In practice, this benefit tends to be outweighed by the risk of having an unbalanced cluster in which some nodes have disproportionately more data than others. For this reason, and because the partitioner is a cluster-wide setting, the default is used in most cases.

While the partitioner generates the token that determines the primary node where each partition is stored, we still need to determine where the additional replicas are stored. This is where replication strategies come in. The two strategies used most frequently are the SimpleStrategy and the NetworkTopologyStrategy. The SimpleStrategy is useful for single data center deployments. You supply a value called the replication factor, which refers to the number of copies (replicas) of your data that will be stored, and Cassandra takes care of the rest. The NetworkTopologyStrategy is used for multi-data center deployments. With this strategy you can specify a replication factor per data center, which is especially useful.

The primary difference of this approach with other NoSQL databases such as MongoDB is that each replica is treated equally, rather than having a primary and backup copies. The flexibility of the platform is also really important. The developers of Cassandra have done a great thing by providing pluggable APIs for partitioners and replication strategies, as well as several other key algorithms, which empowers the community to extend and improve the platform.

InfoQ: Cassandra supports tunable consistency feature. Can you talk about this feature how it compares with strong and eventual consistency models?

Carpenter: Tuneable consistency is an extremely powerful feature which is not always well understood. For example, it is absolutely possible to achieve strong consistency in Cassandra, depending on how you use consistency levels on your reads and writes. The formula “R + W > N” is frequently used to describe how this is done. In this formula, where R and W are the number of nodes that will be read and written, as determined by the consistency level used., and N is the replication factor. The most common way to achieve strong consistency is by using the QUORUM consistency level on both reads and writes, where a quorum is defined as one greater than half the number of nodes (quorum is 2 with a replication factor of 3 nodes, 3 of 4 nodes, 3 of 5 nodes, and so on).

You can use less stringent consistency levels to achieve greater read and/or write speed if you’re willing to allow for eventual consistency. With lower consistency levels there is the possibility that you might not read the latest data for some short period after a write if the read accesses replica nodes that don’t have the latest data yet. For example, to ingest sensor data as fast as possible in an IoT application, you might use the consistency level ONE or even ANY, which returns as soon as any node captures the write in its commit log. This might be perfectly acceptable, especially if your application is not reading the sensor data from Cassandra in real time.

Because each individual invocation of a Cassandra query has its own consistency level, you have a lot of flexibility. For example, if I’m using Cassandra to track some type of inventory, I might increase the consistency on my queries as I get close to selling the last few items to make sure I don’t oversell.

InfoQ: Does Cassandra support storing the image files in the database? What are the design considerations developers need to keep in mind when managing images and other binary data in Cassandra?

Carpenter: Storage of binary data is definitely a use case which Cassandra supports. For example, Globo TV in Brazil provides Globo Play, a streaming video system with DVR-like features which featured at the 2014 FIFA World Cup and multiple national elections. Globo engineers have posted several articles and presentations on the solution they built on top of Cassandra, which I’ll summarize briefly.

A key element to successfully storing binary data is making sure the file sizes do not get too large. A recommended design technique is to break large binary objects into chunks which can then be allocated to separate partitions. Using a fixed chunk size keeps the partitions of equal size and helps ensure a balanced cluster.

Whether this sort approach works well in your situation depends on the access patterns you need to support. In Globo’s case, the DVR features supported by their client application, such as the ability to pause video and resume at a later time or jump directly to a desired time, really lent themselves to this type of solution.

Of course, if you don’t need the flexibility of accessing portions of the binary content quickly, a simpler design approach is to keep the binary data in an external file store such as an Amazon S3 bucket. Then you can use Cassandra to store and search metadata about each binary file.

InfoQ: What features does Cassandra provide in the areas of security and monitoring?

Carpenter: Cassandra provides a rich set of metrics and controls via the Java Management Extensions (JMX), which are accessible via standard JMX clients such as JConsole. Nodetool is Cassandra’s standard command-line tool, which also leverages JMX to support a variety of monitoring and maintenance tasks on individual nodes.

To support monitoring at the cluster level, there are a couple of options. If you’re using the DataStax Enterprise (DSE) distribution, OpsCenter provides some very nice visualizations of cluster status and metrics, as well as automation of key operations such as repair and backup. If you’re more of an OSS shop and running the Apache Cassandra distribution, I recommend extracting metrics from Cassandra nodes via JMX and feeding into a metrics database such as KairosDB (which is itself built on Cassandra), where you can then build dashboards in a tool like Grafana. We’ve had a lot of success with this approach, as the flexibility has been key in identifying key metrics for our environment.

A lack of security features was a weakness of many of the early NoSQL databases, Cassandra included. However, the situation has really changed for the better over the past couple of years. Cassandra now supports authentication and encryption for client-node and node-node communications. In addition, you can apply role based access at the keyspace or table level.  File-level encryption for Apache Cassandra is a work in progress, as there are multiple file types to address (SSTables, commit logs, hints, and indexes), but is a feature provided by DSE.

InfoQ: Can you discuss the Cassandra cluster across multiple data centers and what are the advantages and limitations of Cassandra clusters?

Carpenter: One of the major advantages of Cassandra clusters is that distribution of data across multiple data centers is part of the core design, whereas other databases frequently rely on back-end replication mechanisms which are grafted on later. That being said, the performance of operations such as repairs does require careful planning and tuning. It is important when building clusters across multiple data centers to consider the networking implications across multiple data centers when configuring timeouts. I would always recommend having a private, high speed connection between data centers if your budget allows. You’ll also want to think about the tradeoffs when designing data access in your applications of local vs. cluster wide consistency levels (i.e. LOCAL_QUORUM vs. QUORUM).  

InfoQ: What are some development tools that can help with the developer productivity when working on Cassandra based applications?

Carpenter: There are several tools that developers should have in their toolbox to help with data modeling, application development and testing.

In my experience, most of the key make-or-break choices are made in your data model, before you ever write a line of application code. That’s why it’s very important to have a solid understanding of Cassandra data modeling principles. It’s also why we published the data modeling chapter from the book for free at the O’Reilly website.

In terms of support for data modeling and experimenting with various approaches, I recommend DataStax DevCenter. DevCenter is a free tool that allows you to design schemas and run queries on live clusters. It’s built on the Eclipse IDE platform, so its layout and syntax highlighting should be familiar to most developers. While Cassandra’s CQL shell (cqlsh) provides a great interactive shell experience, I’ve found DevCenter to be a big productivity aid with its syntax highlighting and ability to save common queries. It also traces all of your queries by default and provides a nice tabular report to help you analyze the traces. This is really important for helping to educate developers on what Cassandra is doing behind the scenes, and helps developers learn to avoid anti-patterns such as multi-partition queries that can lead to poor performance.

It can also be helpful to get an idea of how your data models will perform at scale before you invest too much in a particular design. I recommend using the Cassandra-stress tool that comes with Cassandra to generate a simulated read and write load. The caveat is that the tool does not yet support some of the more complex CQL elements such as User Defined Types (UDTs), but you can still get useful results by working around this.

In terms of writing application code, you’ll want to have a modern driver. Historically there were a number of Cassandra client drivers in various languages, developed by different authors and with different feature sets. In the past couple of years, DataStax’s open source client drivers have really come to the forefront, providing support for many common languages for both Apache Cassandra and DSE distributions.

To help with testing, The Cassandra Cluster Manager (ccm) is a great tool implemented in Python that you can use to run a small cluster on your local machine with minimal setup. This is very useful for unit testing and for experimenting with different configuration settings.

InfoQ: Can you talk about Cassandra integration with technologies like Spark, ElasticSearch, and Solr?

Carpenter: We’ve observed a continuing trend toward more complex architectures with polyglot persistence; that is, building systems that incorporating various databases depending on the needs of each kind of data. Spark is really gaining a lot of traction as a technology that can integrate data from a number of sources, including Cassandra and others. Datastax Enterprise (DSE) provides integrations with Spark, Hadoop and SOLR out of the box, or you can build similar integrations yourself.

An interesting use case that I've been exploring recently is using Spark on top of Cassandra, ElasticSearch, and Kairos DB. We can implement some interesting real-time analytic jobs to learn about system behavior by fusing operational data with logging and metrics data. This is just one example of a real-time analytics type of use case that is made possible through integrating multiple Big Data technologies, of course there are many others.

InfoQ: What do you think about using Cassandra NoSQL database with other emerging trends like microservices and container technologies like Docker?

Carpenter: One of the key principles of a microservice architecture is that each service does one thing. A corollary is that each service should have exclusive ownership over its data. There are various approaches to enforcing this ownership. For example, large organizations such as Netflix have gone so far as to create a separate cluster per service, in cases where the scale makes sense. Most of us aren’t building systems that are quite that large, and sharing a cluster between multiple microservices is a more realistic, cost effective approach. If that is your situation, I’d advocate using a separate keyspace per service, and to use Cassandra’s role based access control features to help ensure that only that service has access to its keyspace (aside from administrative access). In cases where you have the need to coordinate changes across multiple data types, you can create additional microservices to compose the microservices that manage those data types, or use an asynchronous-style architecture to orchestrate the changes.

In terms of containerized deployments, I think there are some interesting challenges to overcome in terms of networking, but progress is being made. I tend to think of containerized Cassandra as more appropriate for development environments where you want to be able to bring up and tear down a cluster quickly, with perhaps a little bit of data loading. If you don’t intend the data in your containerized cluster to be ephemeral, then I’d recommend using external storage for your commit log and data files.

InfoQ: What are the features that are currently not available in Cassandra but you would like to see in the future releases?

Carpenter: I’m actually quite satisfied with the current feature set that is available and am just starting to take advantage of the 3.0 features. There are various places that we run into where queries that would be supported in SQL don’t work in CQL, but there are adequate workarounds for these.

I think the larger issue is the learning curve for configuring Cassandra and keeping it running. The most recent ThoughtWorks Technology Radar included a brief statement which I found interesting. They actually recommended steering away from Cassandra, due to its operational complexity, unless you require the ability to scale to a hundred or more nodes. I wouldn’t go as far as that recommendation, but I sympathize with the frustration behind it and believe there is definitely room for improvement. For example, it would be helpful to have some wizards that can help guide the processes of configuring, monitoring and tuning. The out of the box configuration is pretty sensible, but perhaps we can develop configuration templates for a wider range of common deployment patterns.

I do see a lot of promising work in the community in the creation of open source tools to automate some of these complex operational tasks. For example, repair is an important part of how Cassandra maintains consistent data. Repair runs as a background task on a node and there several options for how to run it, which can be confusing to new users. The Cassandra Reaper is a tool which automates repairs across a cluster. The original version was by Spotify, and there is also a fork available. I’d eventually like to see tools like this incorporated into the Apache distribution.

Another trend that is helping to mitigate some of that operational complexity is the emergence of providers that are offering Cassandra as a service. Companies such as Instaclustr and DataScale (recently acquired by DataStax) are offering a variety of hosting and management options for cloud-based, on premises and hybrid deployments. Having someone else take responsibility for managing your clusters can be a really attractive option, especially for smaller organizations and startups that need to focus on creating business value.

Jeff also spoke about the community support for Cassandra database.

Cassandra continues to be one of the most active Apache projects, and has a passionate developer community. One of the most encouraging developments of the past few months has been how the broader community has stepped up as DataStax has begun to temper its involvement in the management and implementation of the Apache project. This includes everything from small consulting firms such as The Last Pickle, to industry giants like Apple and Netflix. The diversity of contributors that we're seeing in both the open source community and in terms of commercial offerings is a testament to both the continued importance of Cassandra and its versatility.

About the Interviewee

Jeff Carpenter is a software and systems architect with experience in the hospitality and defense industries, it. Jeff cut his teeth as an architect in the early days of Service-Oriented Architecture (SOA) and has worked on projects ranging from a complex battle planning system in an austere network environment, to a cloud-based hotel reservation system. Jeff is passionate about projects and technologies that change industries, helping troubled projects find architectural solutions, and mentoring other architects and developers.

Rate this Article