In the past we've had several people discuss the challenges they have encountered when developing microservices, such as Fred George or Dustin Huptas and Andreas Schmidt. Well recently Usman Ismail took part in a panel discussion about the challenges of continuous delivery of microservices and decided to subsquently discuss some of the highlights. He starts by discussing the downside of one of the basic tenets of microservices, that it allows larger teams to move (software) in a more agile manner with rapid prototyping and iteration:
However, micro-services put a significant operational and tooling overhead on your development teams. Each of your services requires a deployment pipelines, a monitoring system, automated alerts, on-call rotations and so forth. All of this overhead is justified for large teams as the payoff from added productivity of feature work is worth the effort of creating these systems. However, in small teams if the same few people are responsible for all the services anyway replicating the pipelines for multiple project is wasted overhead.
Next up is considering the operations overhead. In Usman's view, working in a microservices or monolithic architecture, pushing out a bad version of a service or component requires you to roll back the system and if you get resource constrained you can (often) scale horizontally. However, with microservices you need a lot more monitoring and automation to detect which service(s) needs to be rolled back, determine the impact of rolling back a service on other dependent services etc.
If you have automated alerting (which you really should) then we need to ascribe alerts to service owners and then maintain on-call schedules for multiple services. In a small organization there will be a lot of overlap in the sets of people responsible for each service. This means we will have to coordinate the schedules for these services to make sure the same person is not on the hook for too many services and that people get some respite from on-call rotations. For these reasons as well as those mentioned in the previous section, its better to have a monolith and get all your operational ducks in a row before you start adding the overhead of multiple micro-services into the mix.
Then of course there's the fundamental aspect of microservices which may not be present much, if at all, in a traditional monilithic application: distribution. Which leads to the age old problem of debugging in a distributed environment, something which we've covered many times in the past, and before the term microservice was coined. Previously Bryan Cantrill, CTO of Joyent, discussed debugging microservices in production. Usman believes that not having centralised logging for your microservices is a showstopper:
Furthermore, at large scale having separate monitoring systems (such as datadog, grahite) and separate log aggregation systems (such as ELK, loggly or Splunk) is not feasible. At this scale visualizing metrics and log data is a big-data problem that you are better of solving in one place.
Finally in his highlights of the panel session, Usman touches on deployment coordination and version management. According to the article, one of the big differences between microservices and monolithic applications is that the former is a dependency tree of services whereas the latter is a graph:
For example a typical service stack in the monolithic model may consist of a web array, which calls a cache layer, database layer and maybe a few stand-alone services such as authentication etc. In the micro-services model you will have an interconnected graph or network of services each of which depends on several others. It is very important to ensure that this graph remains a Directed Acyclic Graph (DAG) otherwise you will be in dependency hell and potentially have distributed stack overflow errors.
Interestingly Usman's article then goes on to summarise some guidelines from the panelists. Well, sort of:
No one in the panel was comfortable enough with their micro-service system to propose anything like a guide for how to go about building such a system however, we did come up with some rules of thumb or general guidelines including the following.
These can be summarised here as:
- Microservices require a lot of infrastructure to develop and deploy, so use a platform. "Kubernetes, Swarm, Mesos and their ilk will get you lot of the way there but you still need to unify your monitoring, debugging, continuous pipelines and service discovery mechanisms."
- To ensure repeatability and reliable automation, don't let anything in your system be defined by human processes. "Everything must be defined in code, testable and repeatable. For example, your server/VM setup should be orchestrated using docker-machine, puppet, ansible etc."
- Develop or use centalised monitoring, logging and alerting. "While a monolithic service is like a beloved pet, you know all of its quirks and habits, micro-services are like cattle; you need all of them to be more or less identical, and managed as a generic herd rather than an individual."
- Enforce backwards and forwards compatibility.
- Visualise large microservices deployments as a network. "Monitoring and managing a large micro-services deployment is very similar to managing a network system."
All of the recommendations are based on the collective experiences of the panel members who come from a variety of companies. And those recommendations have evolved over time and will likely continue to evolve as further experience is gained. In some ways one of Usman's final comments echos what Vijay Alagarasan had to say in his talk on Seven Microsrvices Anti-patterns last year:
[microservices] are not a magic bullet to solve the fundamental problems of building and running distributed software at scale. A micro-services architecture does force you to be more conscientious about following best practices and automating workflows. The big take away from the discussion is that unless you are willing to divert a lot of time and resources from feature work into building and maintaining a service framework, its better avoid taking the plunge into the micro-services world. If however, you can invest the time to build a great service framework and workflow then you will come out of the transition as a more agile and productive organization.
We're always on the lookout for further experiences with microservices, both good and bad so perhaps you might want to comment on Usman's efforts or discussion your own?