Virtual Panel: Microservices in Practice
- Understand some of the lessons learned in the past few years, and real-world development with microservices.
- Understand whether the principles for using microservices for brownfield development are the same as when using them in greenfield development.
- Hear from experienced practicioners about some of the latest open source technologies, problems and approaches shaping microservices.
- Learn some of the best practices (do's and don'ts) for using microservices effectively.
- Understand some important considerations before using microservices, such as how they tie into (classic) distributed systems theory and practice.
- Learn whether specific programming languages or technoligies are recommended for developing with microservices.
- Understand whether REST/HTTP should continue as the de facto standard for communication with and between microservices.
Microservices have gone from internal development practices for the select few so-called “Unicorns”, to something many developers in a wider range of organisations are embracing, or considering for their next project. Some people believe that in order to deliver on the benefits of DevOps, microservices are a necessary requirement.
In the last few years we have seen new technologies and experiences shape microservices, often reinforcing their ties to Service Oriented Architectures at the same time as expanding on their differences. Some believe that technologies and methodologies which can assist with developing and adopting microservices are ineffective without associated changes within the organisations that wish to use them.
InfoQ spoke with five panelists to get different perspectives on the current state of the art with microservices, how they are likely to evolve, and to share their experiences, both good and bad, when developing with them.
- Chris Richardson - developer & architect, Java Champion, author of POJOs in Action
- James Lewis - member of the ThoughtWorks Technical Advisory Board
- Martijn Verburg - CEO and co-founder of jClarity
- Christian Posta - Principal Architect at Red Hat, author of Microservices for Java Developers
- Adam Bien - consultant and Java (SE/EE/FX) enthusiast
InfoQ: We are a couple of years into the popularity of microservices; what important lessons have we learned in that time that perhaps weren’t apparent at the start?
Chris Richardson: ‘Microservices’ is a terrible term. It places excessive emphasize on size and leads developers to create services that are too fine-grained, e.g. single REST endpoint per service. Not only that, but the term suggests that it makes sense to have a microservice. For example, I’ve heard “we can do that with a microservice”. I’ve also seen an increasing number of “Xyz microservice frameworks”, which in reality have very little to do with the microservice architecture, e.g. they are simply web frameworks.
It is important to remember that the proper term is the “microservice architecture”. It is an architectural style that structures a system as a set of collaborating services that are organized around business capabilities.
James Lewis: I think the first thing is that widespread adoption has led to a certain amount of semantic diffusion. Martin Fowler and I were fairly clear when we wrote the definition that all of the characteristics we mentioned contributed to the success of the companies using the style. I speak to a lot of big organisations who want to adopt the first characteristic, componentization via services, but who aren't at all keen on the organisational changes implied by the other characteristics. Specifically, Products not Projects, Organised around Business Capabilities and Decentralised Governance. Personally I think that the organisational aspects of microservices are a key success factor for adoption.
Martijn Verburg: Oh there are so many! But I'll pick out some of my favourites:
- Service discovery - Both at development time and at runtime is much harder to solve than people have realized. I've seen many cases where a team of developers is arguing about "where the message went next".
- Distributed tracing - of business logic / transactions is also very difficult to do in a light weight and unified manner. For example, how do you insert a trace that may follow a piece of business logic that travels through 10+ services, all who are built with different technologies?
- Distributed architectures - Microservices based applications tend to lend themselves to distributed architectures, horizontal scaling and load balancing. This is a skill set that traditional monolith developers and sysadmins do not have and must learn. For example, how do you load balance Websocket connections which by their nature are two-way and 'permanent'?
Christian Posta: With respect to the microservices hype, I'm hoping we've learned that there are no utopian architectures; simply adopting buzz-word technology doesn't equate to microservices, and the communication structures of your organization have more to do with the limitations or advantages or your services architecture than previously acknowledged.
I think it's also key that with a microservices-like architecture we've stretched deeper into distributed systems theory and practices that we've studied, implemented and from which we've learned for the last 40 years and that very little is new. What's new is bringing this body of study and implementation to mainstream to solve new business problems.
Adam Bien: I spend most of my time in Java EE projects. The availability of Java EE 6 in 2009 became the main driver towards microservice-like architectures. We were able to package the pure business logic in a WAR and ship it [Java EE 6 Kills The WAR Bloat]. Back in 2009 we called our projects "shared nothing architecture". The term "microservices" had not arrived yet.
Monitoring and performing stress tests was a hard sell back then. With microservices it is more acceptable now to focus on stress tests, system tests and monitoring of the essential use cases.
Prior to the microservice hype, most of the projects focused on the implementation of unit tests only to achieve high (but meaningless) code coverage. This begins slowly to change.
The biggest difference between the early Java EE 6 projects and the current development is the nature of communication protocols. In the 2009 timeframe we mostly relied on binary RPC protocols; right now JAX-RS (REST/HTTP) and WebSockets are the new default.
InfoQ: At the start, the so-called “unicorns” were popularising microservices; do you think this is still the case and if not, then who are the poster children at the moment?
Richardson: Yes. It feels like the majority of the conference talks on the microservice architecture are “cool microservice architecture topic at Netflix/Uber/Slack/Twitter.…”. On the one hand, these talks are incredibly useful and have helped evangelize the architecture. On the other hand, that has led some developers to think that the microservice architecture is a way to address application scaling issues, where in reality it is a way to tackle complexity. In general, it would be great to hear from mainstream companies about how they are using microservices.
Lewis: I think they are still at the cutting edge, yes. It's where a lot of advances are being made in the infrastructure supporting microservices. It is not a "101" architectural style, and some of the benefits manifest most obviously when you are operating at extremes of scale.
Verburg: They still are. Companies like: the BBC, Netflix, Twitter, Amazon et al are all microservice based because they had the horizontal scalability requirement thumped firmly on their desk. But *this* is the major question that most IT organisations fail to address when they blindly jump on the bandwagon. "Do we actually need microservices? Does our scale require it? Does our business logic require it?" The answer for many organisations should actually be a resounding "no".
Posta: To paraphrase Dr. Branden Williams, "there are no unicorns or horses anymore, just thoroughbreds and horses heading to the glue factory"... the internet companies may have been the vanguard showing the way, but I think there are good examples in the traditional enterprise space (FSI, Manufacturing, Retail, etc.) demonstrating the ability to move fast and innovate using technology.
Bien: At the start no one knew what "micro" actually meant. There was a pointless debate about a typical size of service. In Java EE a microservice is a Thin WAR created by a one-pizza team -- two-pizza teams are already too large :-)
In my eyes, the poster children are many enterprise projects based on pragmatic microservices. Unicorns come and go. It is really hard to estimate their success, if they barely survive the first year. Enterprise projects have to last longer.
InfoQ: Some vendors push microservices for greenfield development whereas others tend to focus on brownfield and decomposing monolithic applications; do you think the same principles apply to architects and developers for each approach?
Richardson: The microservice architecture is potentially applicable to both greenfield and brownfield applications that are (or will be) large and complex. What matters is the context within which you are developing the application. For example, if you are a startup still trying to figure out your business model then it is possible that you will be able to pivot more rapidly with a monolithic architecture.
I think that many (perhaps the majority) of business critical applications are large, complex monoliths. The business is in monolithic hell and unable to innovate rapidly. The solution is to incrementally refactor to a microservice architecture.
Lewis: Personally I've been involved with teams who have done both. For brownfield it's often a nice approach since it gives you many more options or seams to start containing a previous system. For greenfield, The approach I favour is to consider the functional and cross-functional requirements, and the context within which the system should run. Sometimes that means a microservice architecture, sometimes not.
Verburg: Same principles but a lot of different compromises :-). Decomposing a monolith is an admirable and satisfying goal to complete, but for a good chunk of that application's life, it's going to be a microservice and monolith hybrid (some of us call this the software incarnation of Cthulu). For example, microservice purists working on a database centric monolith would have put aside their principles and "Do that message passing through the stored procedure in the monolith database" for a period of time until they refactored everything out.
Integration test writing and maintenance becomes very important here.
Posta: The principles are similar insofar both approaches try to find the right boundaries to effect the speed of development cycles. You may have pockets of greenfield development but the harder part is finding the right seams for existing brownfield systems to expand, speed up, innovate, etc and do so safely.
Bien: The vast majority of all client inquiries in 2016 were about the introduction of microservices with the big hope of increased maintainability and cost savings. The problem was never the lack of distribution, rather cargo cult practices and unnecessary pattern implementation.
Splitting a bloated monolith into smaller overcomplicated monoliths will only make the situation worse. In brownfield projects it is crucial to remove the cargo cult based patterns first, and re-learn the domain concepts. Splitting a lean monolith into independent units becomes a fully optional task.
In greenfield projects you should completely focus on business logic and stay with a monolith in the first iterations. Introduce a microservice only if you can clearly explain the benefits. Shipping a lean monolith is still the easiest possible approach.
Both greenfield as well as brownfield share the laser focus on business logic.
InfoQ: What are your top five do’s and top five don’t’s where microservices are concerned?
Richardson: The most important thing to remember is that the microservice architecture is not a silver bullet. You need to carefully evaluate the trade-offs to determine whether it is appropriate for your application.
- Monitor, monitor, monitor.
- Get good at deploying services independently.
- Prefer rapid remediation and canary deploys over integration testing.
- Prefer choreography over orchestration.
- Limit your call tree. The more services in the graph, the more difficult it is to stay available.
- Don't suddenly build 500 services - start with a reasonable number that is supportable with your current infrastructure.
- Don't think they are a magic bullet. You need to understand some non-trivial distributed computing concepts to get good at building them.
- Don't fall for vendor snake-oil - that's why SOA originally died a death.
- Don't forget the bit about replaceability. They should be small enough to be thrown away.
- DON'T DO DISTRIBUTED TRANSACTIONS.
- Did I mention don't do distributed transactions?
- Make sure your team is working in an 'a' Agile manner.
- Make sure your team has a DevOps culture.
- Build three prototype services that communicate with each other and figure out how to do all of the non functional requirements like security, service discovery, health monitoring, back pressure, failover etc., *before* you go and build the rest.
- Let the engineers pick the right technology for each service; this is a major advantage.
- Care more about your Integration tests.
- Start using them because Netflix is.
- Forget about data consistency. "Oh yeah, our microservice architecture doesn't do ACID transactions, sorry we lost your money" is not acceptable.
- Ignore the infrastructure requirements, even for the 'developer desktop'. Get developers mimicking the real PRD architecture as soon as possible.
- Forget about naming - once you've released a public API you are stuck with it. Don't forget to version your API's as well for that matter.
- Throw away the years of developer experience and business logic already written. It's an evolution, not a new paradigm.
- Measure your adoption of a microservices architecture and use that as a guide post. Microservices is about speed so measure how quickly your teams can make changes and deploy without impacting other services. Things like #s of builds, #s of deployments, # of bugs introduced, time it takes to approve a deployment, mean time to recovery, etc.
- Do establish proper feedback loops for your feature teams. It does no good to make changes to your systems/services without knowing what the effect of that change will be. Put developers and feature teams as close to their customer (or even in their customer's shoes) so they see the pain directly from the systems the teams build.
- Do pay attention to data. Data is the lifeblood of a company. When building services, pay attention to use case boundaries, transaction boundaries, consistency issues, and data processing (stream, storage, etc.).
- Build microservices/feature teams with autonomy, responsibility, and freedom built in. Build the tooling, APIs, infrastructure for them to self-service.
- Build your services with instrumentation, metric collection, debuggability, and testing as a first class citizen, not an afterthought.
- Don't just copy the parts of X unicorn company you see just because they seem successful; figure out the principles that drove that company and use that as a guide. Case in point. Simply adopting Netflix OSS technology will not make you Netflix.
- Don't approach microservices as a way to cut costs; Microservices is about enabling innovation and business outcomes through technology not as a way to minimize operating costs, like traditional IT has been treated for decades.
- Don't break down systems arbitrarily small just for the sake of breaking them down. You run the risk of creating a non-scalable, highly inefficient distributed monolith and strangle yourself with transactions.
- Don't ignore the fallacies of distributed systems and the challenges of integration.
- f you have trouble with CI/CD, APIs, DevOps, self-service platforms, self-service teams, then Don't force a microservices architecture; get the principles, practices, and organizational learning systems down first. Doing microservices isn't the goal; fast moving, innovative teams is the goal. Get the foundational pieces in place first.
- Evaluate container technology for deployment. The advantages are too big to be ignored.
- Focus on business logic.
- Monitor the essential use cases / key performance indicators.
- Focus on system tests, not unit tests.
- Automate everything, CI / CD are a no brainer.
- Don't copy the practices of Netflix, Twitter, Facebook or Google unless you have their scale / requirements.
- Do not ignore slow turn-around cycles. Deploying a Fat-WAR takes longer than a thin one. Productivity really matters.
- Do not even attempt to coordinate (XA) transactions between microservices.
- Don't start your project with downloading the internet. Less is more. Each dependency makes your deployment slower and requires security audits and bug-fix maintenance. There are no "free" dependencies.
- Don't distribute (or only distribute with obvious advantages). A lean monolith could become the best possible choice.
InfoQ: HTTP or REST/HTTP is often seen as the de facto standard for communication between microservices, yet we’ve recently seen a lot of groups talking about asynchronous, message-oriented approaches instead; what do you think?
Richardson: Yes. The de facto IPC mechanism these days is HTTP/REST. It is familiar and easy to use. The drawback is that it introduces a temporal coupling between the client and service. Whether or not that is a problem depends on the context. For example, when writing code that handles a query request that aggregates data from multiple services, then it might be ok. If on the other hand, you are handling a command request that updates data, you should use asynchronous messaging to implement eventually consistent transactions, a.k.a sagas.
Lewis: Smarts in the endpoints does not strictly imply REST/HTTP; for me it was about these groups of small collaborating services that encapsulated their own logic, communicating via a uniform interface. I've been involved in teams that have used lightweight messaging and RESTful approaches to supplying that uniform interface and both have been successful. Still, the most important thing is to choose the appropriate patterns for the problem at hand. If you have a business process that lends itself to asynchronicity then you should use an asynchronous integration technique; conversely if your problem is more amenable to map-reduce aggregation over a number of discrete services then you should probably use some form of reactive approach. Once again I'm reminded that we often try and take a reductive approach when actually it's really about thinking for yourself.
Verburg: I think both approaches will be used with asynchronous messaging becoming more popular over time. At jClarity for example, we have an asynchronous message-oriented approach but also offer a REST/HTTP(S) API for easier public consumption.
Posta: I think as you scale out systems like we talk about with microservices, they tend to exhibit characteristics we see in other Complex Adaptive Systems (stock markets, ant colonies, communities), to wit: autonomous agents, independent decision making, learning/adaptation driven by feedback, nonlinear interaction, etc. In systems like that, events, message passing, and time are all critical enablers that tend to look like the "async" model. IMHO making time the focal point between these systems (as well as the fact our communication channels may not be reliable) force us to deal with reality up front and make for a model we know scales in other applications.
Bien: In my projects HTTP / REST (always JAX-RS) was good enough for the realization of the vast majority of all use cases. Sometimes we also introduced WebSockets as an asynchronous, peer-2-peer, messaging protocol. These cases were more an exception than a rule.
InfoQ: Given that microservices architectures are more distributed systems-oriented than some developers have been used to in the past, where should a developer new to microservices and perhaps distributed systems, start?
Richardson: A developer will use many of the familiar frameworks and libraries to develop an individual service, so not much has changed there. However, a consequence of applying the microservice architecture is that some things such as transaction management and querying need to be done differently. A good starting point to learn about those issues and how to address them, are my recent Infoq articles as well as my website.
The essence of the microservice architecture is the idea of organizing services around business capabilities or subdomains (or bounded contexts). I’d recommending reading Eric Evan’s book Domain Driven Design, since both aggregates and his ideas around strategic design are central to the microservice architecture.
Lewis: I think Sam Newman did an excellent job with his book "Building Microservices", so I would start there. For background, reading I would recommend "Domain Driven Design" by Eric Evans, "REST in Practice" by Webber, Robinson and Parastatidis, "Enterprise Integration Patterns" by Hohpe and Woolf, and "Release It!" by Michael Nygard. For a peek at the philosophy behind building systems composed of small things, I heartily recommend "The Art of UNIX Programming" by Eric Raymond. Further resources can be found on Martin Fowler's site too.
Verburg: If they're a traditional Java enterprise developer then the new Microprofile.io community is a great place to start. Regardless of where they start, they *have* to understand what it takes to set up the infrastructure. Start by renting a few Linux boxes (and/or taking the plunge into Docker) and building a "hello world" service that talks to an "Hi Back!" service on the other 'machine'. You should experiment with HTTP(S), certificates, load balancing, IP tables, having a distributed data store (like MongoDB) and so forth.
The application code is now truly the easy part of this new world. The hard part is the plumbing.
Posta: This is such a great question. The last 40 years of distributed systems computing research and practice is the core backbone of implementing a microservices architecture. Understanding why your ACID database is so good for you and the challenges you'll need to overcome when you distribute things is paramount. There are lots of good papers on this. Ones by Jim Gray, Peter Bailis, Alan Fekete, Pat Helland, Leslie Lamport etc are my favorites. My background is in integration and messaging and I've also found those to be a hugely valuable body of knowledge to help set the foundational concepts.
Bien: Just focus on domain concepts, target domain and the users. Keep the signal to noise ratio as high as possible. E.g. the best Java EE projects only contain business logic with a few annotations, without any additional cruft, patterns or indirections.
Forget about all modularization attempts from the past. Keep your code simple. Thin WARs are the ultimate module.
Be paranoid and assume that the whole infrastructure around your service can and will fail. Test the behaviour in failure case. Provide the simplest possible solution (should be classes, not frameworks).
InfoQ: Are there particular languages or technologies you’d recommend for developing with microservices? If so, why? Any that you’d avoid? If so, why?
Richardson: The short answer is that the microservice architecture is independent of languages and frameworks. The longer answer is that some languages might have more microservice chassis frameworks that help with building distributed applications than others. For example, Java developers can use Eureka/Ribbon, possibly via Spring Cloud, for client-side service discovery, and the Hystrix circuit breaker library. On the other hand, there are good arguments for using a deployment platform that provides server-side discovery so that the developer doesn’t have to worry about it.
- Microprofile.io, Vert.x, Spring Boot, JHipster for Java developers. At jClarity we use Vert.x which is an amazing (JVM based) polyglot language library for developing Microservices applications. Can't recommend it enough.
- Akka for Scala developers
Posta: Use whatever you're comfortable with that will help you go fast. In my mind, Go, Java/.NET, NodeJS are the most often languages used for these types of services.
In the enterprise, if you're trying to modernize your Java services, technologies like linux containers and "micro frameworks" like Dropwizard, WildFly Swarm and Spring Boot are helpful. If using Domain Driven design event frameworks and the reactive frameworks like Vert.x are awesome. Other cloud scale technologies for both the platform and the application layer include Kubernetes, Hystrix, and Envoy help solve difficult distributed systems issues.
Bien: Java is 20 years old, mature, and comes with unbeatable tooling and monitoring capabilities. At the very beginning, Java already incorporated microservice concepts with the Jini / JXTA frameworks mixed with no-SQL databases like e.g. JavaSpaces. As often -- Java was just 15 years too early. The market was not ready for the technology back then. However, all the design principles from 1999 still do apply today. We don't have re-invent the wheel.
I frequently suggested Java EE (the full profile) application servers, (Payara, TomEE and Wildfly) as a microservice platform for startups / greenfield projects tipi.camp, artem, dreamit, next farming (...). We started with the realization of business logic in the very first hour without wasting any time for discussion. We were productive from day one. The developers were positively surprised about the development efficiency, memory footprint and built-in features of modern application servers. Developers were stunned at how much you can achieve with stock Java SE / EE without any external library. I got only positive feedback so far.
Docker is another key ingredient to success. Coupled with Java EE it is a dream team.
InfoQ: Where do you think we’ll be two years from now with microservices?
Richardson: Who knows!? I first gave a talk about what is now known as the microservice architecture in April 2012. The core idea of the microservice architecture has remained unchanged. Since then I’ve since seismic shifts in technology: the rise of Docker and AWS Lambda for example. Consequently, it is difficult to make predictions. Having said that, I expect (or hope) that the microservice architecture will traverse the Gartner hype curve and will reach the plateau of productivity.
Lewis: My prediction is that some companies will have made a lot of money by adopting them and there will be a number of organisations that have tried, but not understood the implied organisational changes and who will have gotten into a terrible mess. Also, I hope we get some more really cool tooling around service visualisation, request tracing and more intelligent failure detection. That would be nice.
Verburg: In terms of the Hype Cycle, curve we'll have crested the Hype Wave and fallen into the Trough of Disillusionment; some organisations will be heading towards the slope of enlightenment :-)
Posta: I'd like to reframe the question if you don't mind: given we're about two years into the microservices hype, do you think in two more years we'll be at the same point we were with SOA 4 years into? Yes :)
The difference this time is that the internet companies and startups are significantly disrupting traditional enterprises with technology as the main weapon (the game has changed). So with regards to the technology hype, it'll be no different than SOA, but enterprises that don't adopt the many principles that make up DevOps, Agile, Microservices will lose to those that do.
Bien: During a Java User Group meeting a developer proudly stated: "My 4-devs team ships 35 microservices". Another day a consultant approached me to present 70 JVM instances running on his notebook.
Both were surprised about my question: "Why are you doing this? What is the added value of your services?"
I expect the first exaggerated microservice projects to be more expensive as estimated. Such projects may cause the first articles / conference talks about microservice bloat or low developer productivity to appear on the horizon.
Bad press usually leads to another extreme. I'm not sure whether we get Macroservices or Nanoservices. I'm pretty sure we get another old concept "sold" as new with another funky name.
In this virtual panel article, we learned about the current state of the art with microservices, experience-driven best practices and some predictions for where things may be heading in the next few years. We got various recommendations from the panelists about technologies and approaches that can help with using microservices successfully, but also that you should never lose sight of the fact that a microservices architecture is inherently a distributed system and therefore decades of theory and practice may be hiding beneath the surface waiting to pounce on the unsuspecting developer. We also heard their thoughts on REST/HTTP as a means of communication with and between microservices as compared to other mechanisms such as asynchronous, message-oriented implementations.
About the Panelists
Chris Richardson is a developer and architect. He is a Java Champion and the author of POJOs in Action, which describes how to build enterprise Java applications with frameworks such as Spring and Hibernate. Richardson was also the founder of the original CloudFoundry.com. He consults with organizations to improve how they develop and deploy applications, and is working on his third startup. You can find Richardson on Twitter @crichardson and on Eventuate.
James Lewis studied Astrophysics in the 90’s but got sick of programming in Fortran. As a member of the ThoughtWorks Technical Advisory Board, the group that creates the ThoughtWorks Technology Radar, he contributes to industry adoption of open source and other tools, techniques, platforms and languages. For the last few years he has been working as a coding architect on projects built using microservices; exploring new patterns and ways of working as he goes. He rather likes the fact that he got to describe his take on things jointly with Martin Fowler in an article that is influencing how people see the future of software architecture. In short, he’s a bit of a loudmouth on architecture, programming, organisational design, lean product development and basically anything else that’ll improve the way he, and others, work. He also quite likes Lego, Warhammer 40K and working for ThoughtWorks. Lewis is co-author of Microservices: a Definition of the Architectural Style.
Martijn Verburg is the CEO and co-founder of jClarity, a Machine Learning based Java/JVM performance analysis company. He is the co-leader of the London Java User Group (LJC), and leads the global Adopt a JSR and Adopt OpenJDK efforts to enable the community to contribute to Java standards and OpenJDK. He is a popular speaker at major conferences (JavaOne, JFokus, OSCON, Devoxx etc) where he is known for challenging the industry status quo as "the Diabolical Developer". Verburg was recently made a Java Champion in recognition for his contribution to the Java ecosystem.
Christian Posta (@christianposta) is a Principal Architect at Red Hat and well-known for being an author (Microservices for Java Developers, O’Reilly 2016), frequent blogger, speaker, open-source enthusiast and committer on Apache ActiveMQ, Apache Camel, Fabric8 and others. Posta has spent time at web-scale companies and now helps companies creating and deploying large-scale distributed architectures - many of what are now called Microservices based. He enjoys mentoring, training and leading teams to be successful with distributed systems concepts, microservices, devops, and cloud-native application design.