Using Kanban with Overbård to Manage Development of Red Hat JBoss EAP

As planning the work for Red Hat JBoss EAP became harder and harder, Red Hat decided to adopt Kanban to make their development process more manageable, while maintaining a very high level of quality. At the Agile Greece Summit 2019, Dimitris Andreadis presented how they introduced Kanban in their distributed team and explained why they developed their own Jira add-on for visualizing the work, and how by adding parallel tasks to their Kanban cards they simplified the workflow.

The EAP team started with Kanban by mapping the full details of their workflow onto Kanban states. Visualizing the states on the Kanban board allowed everyone to have the same view, said Andreadis. Using a continuous process they optimized their process by adding, combining or removing states.

Having a complex process with many states made it difficult to visualize the work using standard Jira plugins. They decided to develop their own plugin, called Overbård, which enables them to have everything on a single board and uses filtering to reveal the information that interests them. This plugin also makes it possible to create new views on the spot, said Andreadis.

Andreadis mentioned that with Kanban, you can totally respect your existing process, roles and team dynamics, capture and visualize work in progress, then iterate and continuously improve on it. It puts you into a continuous self-improvement mentality.

InfoQ interviewed Dimitris Andreadis, director of engineering at Red Hat, after his talk at the Agile Greece Summit 2019 about the problems that they were facing, why they decided on Kanban over Scrum, how they introduced Kanban, how the Jira add-on Overbård helps them to view the ongoing work, using parallel tasks in Kanban, and the benefits that Kanban has brought them.

InfoQ: What problems were you facing?

Dimitris Andreadis: Red Hat JBoss EAP (Enterprise Application Platform) has become a very complex product. As a result, planning EAP releases is also increasingly complicated. In one extreme case of the team working on the next major release while developing features for the previous minor release, the planning for that major release was ongoing for 14 months with the requirements constantly changing. However, spending more effort on planning didn't improve the end result; it didn't make us any smarter or more accurate. We'd rather spend more time doing stuff rather than talking about it. That was a major problem.

In addition, there were cases in which requirements could be misunderstood or miscommunicated and we found that out late in the cycle. We had to find a way to collectively iterate over a requirement and make sure everyone understood what was to be done. In some cases we could go as far as implementing a proof-of-concept before we would be certain we fully understood the problem and the proposed solution.

Finally, tracking the progress of a release in such a big and distributed project was hard. There was too much parallel action going on from different sub teams working on a multitude of components. There was no easy way to have a consistent view of the current status; a single place to point someone to track progress. Naturally, sticking to agreed release dates was even harder.

InfoQ: What made you decide to adopt Kanban over Scrum?

Andreadis: We have a large distributed team with small subteams specialized in certain areas, so coordinating all of that using a tight Scrum setup seemed difficult and very costly. We'd need to create six or more Scrum teams and a Scrum of Scrums on top just to get the process going, meaning we had to find or create an equivalent number of Scrum masters as well as invent a way to clone our unique product manager into many product owners.

The subteams are made of highly specialized and often veteran developers working on their own upstream communities with their own processes that have evolved over the years. We didn't want to alter their tried and tested processes. And we would have to squeeze in with the same Scrum team people working on different areas. For example, people working on security and clustering have little in common; there was no good reason to put them on the same Scrum team.

Finally, a lot of complex features would often require weeks or even months of development, and trying to micromanage this in short sprints didn't make sense. There was no obvious way to break those features in meaningful pieces, neither a "customer" who could test those intermediate deliverables. And when it comes to implementing specifications (like Java EE 8), you are by definition feature boxed. You are done when you are done.

InfoQ: How did you introduce Kanban for EAP?

Andreadis: Gradually! The very first step is to recognize your actual process and try to map it onto Kanban states. And when we talk about states, we mean *all* the states. Not just a simplified ToDo, Doing, Done. You want to do what I call deep Kanban by capturing the full details of your workflow. This is easier said than done. It is quite common for complex products developed over multiple years to have the process embedded into people's minds. A big part of it might be tribal knowledge. So what you find initially you may not like very much, but that's not a problem of Kanban, that's the reality.

After doing that you can visualize your states on the Kanban board so that everyone gets to see the same view. It might be challenging depending on the choice of tool, but it's important that you use a tool tightly integrated with your ticket tracker, JIRA, github or otherwise. You want to have a single source of truth. If you think about it, the board is just a visualization of your work.

Finally, after having accurately mapped your initial state, you can start thinking about ways to optimize it. For example, we've introduced a new analysis phase to make sure Product Management/Engineering/QA all agree on the interpretation of a requirement. In another case, we've allowed a more liberal transition from state to state to enhance parallelism in the workflow.

This is pretty much a continuous process so it's important to go step by step and not rush things, because you don't want to scare off your people or make their lives unnecessarily complicated. Trying to explain at every point what problem you are trying to solve and agreeing with solutions with the team is very important because they are the ones who will use this process, so it's important to have buy-in from everyone.

InfoQ: You developed a Jira add-on called Overbård. Can you elaborate what it does and how people can get it?

Andreadis: Soon after mapping our workflow onto Kanban states we've hit a wall because we had too many of them (around 23) and we just couldn't find a convenient way to visualize this using the standard JIRA Agile plugin. You couldn't even see the states on the screen; they were all squeezed in. We've looked at other tools but we didn't want to have to synchronize state between multiple systems and we wanted to have a single source of truth, our JIRA tracker.

We also wanted to have everything on a single board and use filtering to reveal the information that interests us, rather than having many linked boards to maintain. And filters and swimlanes were very static in JIRA; they had to be setup by hand, and in our case we'd like something more dynamic because we had too many of them and we wanted to be able to create new views on the spot.

After failing to find something appropriate for our requirements, Kabir Khan, WildFly/EAP Core developer who has also (very successfully) played the role of a release coordinator, came up with the idea of developing a custom plugin for JIRA that could solve those problems. So he went on and created Jibran, later renamed to and open sourced as Overbård (note: the name is Norwegian). Overbård is essentially a viewer on top of otherwise normal JIRA projects. It uses some limited configuration to know where it should pull the information from and how to visualize it in order to create the boards. Except for that, all other states are pretty much pulled from the JIRA issues themselves, so there is no magic there.

Overbård can deal with many states by supporting horizontal scrolling & collapsible states, or groups of states. It’s completely dynamic and supports arbitrary filtering with zero configuration. Any views you create are bookmarkable so you can return to them very easily. And it’s super fast because it caches data on the server. It essentially loads the project issues in the cache and then filtering becomes instant. Overbård really saved the day for us and facilitated our agile transformation journey.

You can get Overbård for free from here.

You can also play with a read-only version of a demo Overbård board here.

InfoQ: How did you add parallel tasks to your Kanban flow?

Andreadis: Relatively soon we've realized that Kanban states are very sequential, whereas the reality is much more parallel. Quite often you have multiple people collaborating on a feature, e.g. developers, testers, and documentation writers, and everyone might be at their own stage in the overall workflow. Mapping all those on Kanban states can be challenging and can lead to an explosion of states or the introduction of pseudo-states that however fail to capture reality, while making the workflow unnecessarily complicated.

One way to deal with this is to introduce some sort of linked tasks, but this produces a plethora of interconnected tickets which makes things unnecessarily complicated and hard to manage.

So we came up with the idea of introducing new custom fields on the JIRA ticket to capture the various parallel tasks of interest, for example, one of them could be Test Development (TD) a task that goes on in parallel with the main feature development and that in turn can have its own states, likely different from the main Kanban card it belongs to. With the parallel tasks we essentially introduced a third dimension to the two-dimensional Kanban board.

Here's an example of a Kanban Card with the parallel tasks marked with their initials at the bottom row and with respective color coding to reveal their current state. For example, TD stands for Test Development for the feature described on the card and can take the states ToDo/Red, InProgress/Yellow, Review/LightGreen, Done/BrightGreen. Every parallel task can have their own independent set of states, so for example, Product Documentation (PD) comes with seven different states. Parallel task states can be changed directly on the card and pretty much everything on the card is supported by tooltips with additional information.

Parallel tasks is a very powerful concept and it has multiple benefits:

• It creates a much more compact view of the board, with a lot less interconnected tickets.
• It simplifies the overall workflow, because you need less states on the main Kanban card
• It captures much better the reality of development. There are multiple things that happen at the same time and each of them can be at different stages at any given moment.
• It lets you create with a few clicks arbitrary queries to reveal, for example, developed & tested features whose docs are not verified yet.
• You could use the parallel tasks to track various merging pre-conditions, for example, when all the parallel tasks go green (you can color code the states) then we are ready to merge to the mainline or deploy to production.

Now, since we controlled the tooling (Overbård), we could enhance it to understand and visualize those parallel tasks properly, so they were not simple dumb fields on the JIRA ticket, but effective enablers of our process.

InfoQ: What benefits has adopting Kanban brought you?

Andreadis: There are many benefits to adopting Kanban.

First of all, when you start with Kanban you can totally respect your existing process, existing roles and team dynamics. Kanban forces you to accurately model your workflow, how you actually do the job. You need to be very specific about states, inputs, outputs, handoff points, definitions of Done. This is all very important in order to capture and visualize work in progress, then iterate and continuously improve on it. Kanban puts you into this continuous self improvement mentality which is very important.

Kanban fits very well our asynchronous way of work and helps us maintain a very high level of quality for our deliverables. With a well groomed backlog and timeboxed releases we've simplified very much our planning process. We agree on the major themes for a release and then developers are responsible for pulling work, thus reducing context switching and planning time.

Overall we've managed to reduce an 18 month average release cycle down to performing three month iterations. A six times reduction is huge for a project of this complexity. And we've managed to do that with essentially one guy running the show *and* also developing the tooling required to support the workflow. We'd need more than 6-8 people (or more) just to resource the process itself in an equivalent Scrum setup. People find it hard to believe we've achieved so much with so few resources.

I think Kanban is somewhat orthogonal to Scrum. Scrum to me looks more like a box or a working framework describing what goes on in the periphery of the box. It defines processes, roles, responsibilities, ceremonies, release cadence. But it says very little about what goes into the box, about the actual work that is going on. It makes the assumption that if you put everyone on a daily standup they will somehow figure it out. Scrum is very synchronous.

Kanban on the other hand is more like a methodology that you can apply on top of an existing process. Kanban cares very much about what's happening inside the box; the actual work that is going on, how you capture and visualize it, then gradually iterate in order to optimize it and increase flow and thus output. Kanban can fit into different size boxes. You can perfectly do Kanban within Scrum (ScrumBan), plus, it can operate very well asynchronously, providing better scalability.

In any case and no matter what methodology you choose, it's important that you adapt the tools to your process, rather than adapt your process to the limitation of the tools, which is exactly what we did by developing Overbård, our own JIRA plugin. And since that worked for us, it might work for other people, too, so please feel free to try out Overbård and see if it fits your development process.

About the Interviewee

Dimitris Andreadis has 20 years of experience in IT and he is currently the director of engineering at Red Hat in charge of the Quarkus team. Before that he was running the WildFly / JBoss Enterprise Application Server team for several years. He also served as the JBoss AS project lead and he has been a JBoss addict and contributor from the early start-up days. He previously worked at Intracom and Motorola in the areas of NMS/OSS, designing reusable frameworks and distributed systems. ANdreadis studied computer science at the Technological Educational Institute of Athens and received an M.Sc. by research from University College Dublin, Ireland.