Kiki Carter, enterprise architect at Lightbend, presented “Somm” Lagom: Building Systems That Age Like Wine at the 2017 Emerging Technologies for the Enterprise (ETE) conference.
“Somm” is a reference to the 2012 movie of the same name where four wine stewards attempt to pass the “Master Sommelier” exam. Carter used this analogy to discuss how software applications should age like fine wine. She discussed the challenges that enterprise developers face in the new era of rapid system building. These include:
- Too many choices, analysis paralysis
- Difficulty to consistently scale architectural integrity
- Experts required
- Unplanned complexity or chaos
She introduced Lagom, Lightbend’s “opinionated” microservices framework, based on Akka and Play, and discussed how Lagom addresses today’s enterprise challenges. Carter maintained that “rapid change usually means rapid aging.” In summary, she maintained “in order to keep up with the pace of change and maintain architectural integrity as you move, try using a framework that provides an abstraction above the application level - at the system level.”
Carter spoke to InfoQ at ETE with her thoughts on microservices, reactive systems, Scala vs. Java, and the SMACK (Spark, Mesos, Akka, Cassandra, Kafka) stack.
InfoQ: How long have you been with Lightbend and what are your current responsibilities?
Kiki Carter: I joined Lightbend last year in July, so come summer, it will be one year. I worked with Lightbend on the other side of the fence as a customer and client for five years.
I’m an enterprise architect here. A lot of my responsibilities are related to interactions with our larger enterprise companies by helping them better understand our technology, describing when we have new features and software products, being able to evangelize with them, and to grow our usage and adoption within those companies. Maybe they’ve started out with a small project, seen the benefits, and want to expand it, and have some growth. I help out with that as well. Enabling our teams to really grow our reactive technology in the enterprise - that’s really the core of my responsibilities.
InfoQ: Bold Radius develops applications and provides consulting services around Lightbend. OpsClarity develops tools for monitoring reactive systems. Both of these companies were recently acquired by Lightbend. How has the acquisition influenced the way Lightbend offers services to their customers?
Carter: That's a really good question. I'll start with Bold Radius. One of the things that happened with that acquisition is that it enabled us to service a broader range of customers. We still work with other partners who do the same thing as far as professional services. And as you know, we're not a consulting company, but that helped us enable companies to not only use our technology, but also grow that knowledge within their existing organization so that we can move out and let them take over. We focus more on enabling our customers and partners rather than showing up to do all the work, leaving you with a sketchy “hand-over.”
With OpsClarity, that's another interesting side of the coin where we offer our customers even better ways to visualize their telemetry. As you know, monitoring asynchronous and reactive applications and distributed systems is not something that's trivial. And the telemetry that we’ve provided with Lightbend monitoring has been really instrumental in helping people to, at least, monitor those apps. But what OpsClarity brings is another level of that monitoring by providing some very beautiful visualizations for people to have a good bird's eye view of their systems. So the visual aspect, as well as the AI aspect, being able to spot anomaly detection, that's built into OpsClarity. We’ve really taken that telemetry to the next level where enterprise operations teams can really benefit from it and use it in a smarter way.
InfoQ: Microservices has been an interesting topic over the past few years. Based on the Enterprise Development Trend 2016 report, there seems to be an even distribution of organizations (a) using microservices in production, (b) considering using them, (c) experimenting with microservices in a sandbox environment, or (d) no interest at all. There is also plenty of literature warning developers on using microservices including, “The Seven Deadly Sins of Microservices.” The takeaway message seems to be “use microservices with care.” Is there anything Lightbend can do to address those particular concerns?
Carter: Yes, absolutely and I feel like “use microservices with care” is the right word. And when you don’t, what you experience, and what I think happened in industry, is a huge blowback where people use microservices, experience the chaos that happens when you don't use them correctly, and then shrink back - again, resistant to change. But, what we have to address that need is a framework that abstracts at the system buildling level to encode architectural best practices. So some of those mistakes you make or some of the headaches you see in building microservices, using our framework, Lagom, built specifically for microservices, those best practices, design patterns, architectural principles, and even the underlying technologies and tools being used, have been “encoded” in such a way that simply using the Lagom semantics will pretty much guarantee that you're writing microservices the correct way. That's the beauty of Lagom. It solves technical problems, but it’s a thin abstraction over other applications, frameworks, libraries and tools that brings those together to allow you to solve these problems in a consistent way, that is maintaining architectural integrity with the ability to deliver not just a technical asset, but something that speaks to the design of your system as well. That’s really the principle of Lagom - making sure that you’re doing microservices right and trying to help you to avoid the chaos.
InfoQ: What are your thoughts on how microservices should be used?
Carter: I feel microservices should be used to decompose a monolith, not for the sake of decomposing a monolith, but for the sake of allowing you to deliver things more quickly to your clients and users. They need to deliver features more quickly, be able to make your systems more fault tolerant, having isolation and things that enable your systems to get stronger over time, to grow over time, and to age well during the life of your system. Change happens so quickly now that being able to change your system quickly, build your system quickly, and modify it becomes an area where having microservices, and having those managed in a consistent way, helps your system to age with grace.
InfoQ: With regards to decomposing monoliths, would it be fair to say that you should start a new application using microservices?
Carter: Well, yes, but you reuse your monolith parts. I wouldn’t say that in order to use microservices, you need to burn down the system, you have and start over. And when you’re adopting Lagom, you don’t have to do that. You can decompose by feature rather than decomposing by specific technical components. And if you decompose by feature, you pull existing components, use composition, and maybe add other things in order to bring that feature itself into a new service with the ability to interact elegantly as you continue to decompose the existing monolith.
InfoQ: How do you see the SMACK stack becoming more prominent in software development in, say, the next five years?
Carter: That's a really good question and I think the prominence is going to happen even sooner than that. In the next five years, something else will come up, but at least for now, I see the SMACK stack as something that is heavily in use because of what we found as this data economy - the richness of data and being able to do huge amounts, but also quickly. We don't say “big data” at Lightbend, we say “fast data.” It's about being able to gain insights on your systems based on the data present in your systems and act on them quickly before it gets stale. Data gets stale a lot faster these days. If you need to make a quick decision about an ad to serve or a quick decision about a recommendation, you need to be able to do that in real time. And the SMACK stack enables you to do that. The way I see it, in the next five years that stack is going to evolve. Right now, you see huge prominence with Kafka, for example. And maybe in the future, there will be other players in the distributed, commit-log game. You see people like Uber coming up with Cherami, something similar, but not the same as Kafka. I think what's going to happen is that stack is pretty ubiquitous right now because people have pulled it together. These are the best in the field, but I think that this will evolve over time and you'll see more people entering into the stack. I see Spark staying long-term, but I also see Flink and beam increasing in prominence. I think this fast data space is something that will only grow in the future. At some point, we’ll see an explosion of technology that addresses problems. Then at some point, pull it back to a few and say, “we will standardize on these.”
InfoQ: How do you see reactive programming/reactive systems evolving in, say, the next five years?
Carter: I do see some definite shifts in this area. One thing you'll find is, again, at least in the reactive programming space, we have gaps to address. Many times, when people are talking about reactive programming, they’re focusing on asynchronous communication, maybe asynchronous I/O and non-blocking communication. But, when we talk about reactive systems or reactive programming in the Lightbend sense, there's more to it than that. When we look at the Reactive Manifesto which is how people regard this vision of information, it also deals with fault tolerance and elasticity which is very indisposable when it comes to building systems. Maybe you can ignore it for a minute when you’re just doing reactive programming in case you’re building a monolith, a reactive monolith maybe, but once you start weaving systems together, systems that need to be interconnected and work together in concert, you really need to have that fault tolerance and that resilience and elasticity at the systems level. By that, I mean not depending on, for example, a coarse grain disaster recovery type solution using simply containers or VMs or something to say I’ve got resilience; but being able to be resilient more-or-less at the application layer before depending on an external container, or VM, or cloud platform, it's a little bit different.
InfoQ: I read somewhere that applications built with Scala have dropped since Java 8 included functional programming. What would you say to developers considering the choice between Scala and Java to build their applications?
Carter: Actually, the presence of Java 8 may not be the reason why people are using Scala less. There are other things happening around the same time. Maybe some people who were using Scala decided to try out Go or started using JavaScript frameworks a lot more. They aren’t necessarily correlated events. I think that a lot of players on the field who have been pushing this lambda functional programming have come to the space and give people room to try it with familiar languages.
If someone were asking me to decide between Java and Scala, that decision has to do more with your business goals. What are the goals of your company? What are the goals of your team? What are the business goals? For example, if you’re working with a lot of data, you maybe using Spark or you’re working in an environment that requires real data consistency. Working with immutable structures is very natural in Scala. It’s by default. Whereas with Java, you have to do different things in order to make your data structures immutable. While Java may be able to give you things like lambdas, for example, it doesn’t really guarantee that you're getting all the features of functional programming, being no side effects, by having no immutable data structures. In case you don’t want to run behind your team of developers who are used to imperative programming or used to mutating objects rather than creating new objects in place, that becomes a paradigm shift for people who are using Java one way, then decide to use Java in a functional way, without understanding the changes you have to make to your Java coding habits. Whereas if you’re using Scala, you’re doing that already. It really depends on what your goals are. If you’re really looking to make a system that is highly concurrent and has really good data integrity, I would suggest looking at a language that uses immutability as a key characteristic.
That’s just one example. But, I’d go through a list of things that people would prioritize as far as their ultimate goals. It’s always better to invest in something, even if you invest up front, in training or learning, invest where it matters to your end goal. If you invest in learning Scala because it’s going to help you implement specific patterns more easily down the road, that’s a good investment. But to say that we already know Java but then it’s hard for you to implement those patterns as you go on life with your application, that becomes a trade-off you have to think about.
InfoQ: Interesting. And the funny thing is that both languages compile to the JVM.
Carter: And if people are really wavering, do what you’re most comfortable with. Don’t be closed minded. Keep an open mind about your system’s evolution because the great thing about Scala and Java, like you said, compiling to the JVM, is you can start down one direction and change at any point. It all has good interop so you don’t lose anything at the bytecode level.
InfoQ: What else would you like our readers to know about yourself, your work, and/or any other item that I may have missed?
Carter: I love travel and I’ve visited a number of countries. I love aerial views and skydiving views, but I’m afraid of heights. I do things that I’m afraid of because there’s no other way to get the end goal. There’s no other way for me to see a beautiful aerial view in the way I like to see it other than skydiving, so I can’t say that my fear of heights prevents me from doing that. Which is funny because I see that when I’m working with architecture organizations who have moved into “change-resistant” mode. There’s no other way to get the end goal. You have to give it a try and do it.
Other than that, I also like social work. That’s my other life. I like to work with organizations that prevent human trafficking. I work with rescue organizations that help rescue girls who have been sold by their family or kidnapped. That’s another passion of mine. I try to look at the interaction of technology and what we can build in order to solve some of those problems.
Editor’s Note
Michael Redlich has been an active participant in ETE since 2008 as an attendee, a speaker, and more recently, as a member of the ETE Steering Committee since 2013.