A collaborative approach for real-world BPM

We have been working in the business process management (BPM) space for years already, and it is very interesting to see the recently growing attention for it. Catalysts for this interest may be the growing maturity of the tools, the new 2.0 version of the BPMN standard, better understanding caused by more publications or improved preconditions for BPM approaches, to name only a few of the most important developments within BPM.

Vendors offer more and more high-sophisticated graphical tools, which promise automation of business processes without any coding or even developers; however, we see a problem with these "traditional" vendor centric approaches: They don’t deliver on these promises!

Let me give you some examples to support this statement from our previous projects. Without naming the concrete tool, which wouldn’t be fair since most of them share the same basic problems, a colleague had to implement an easy process with a small web GUI. The tool introduced an own magic Drag and Drop GUI designer, which seemed to be handy in the beginning, but when we almost finished the project, there were some small data validation requirements in the form, which the magic tool wasn’t designed for. In an attempt to get around these limitations, we spent more time futzing with the designer than we needed to implement the entire GUI in plain JSF, which we did in the end anyway.

For another customer, one developer told me, "It took more than two days to try to model some special requirements, which he could have written in Java directly in half an hour!" Another customer tried to get transactions and stateful services running, which unfortunately required calling services as Web services. After experimenting with WS-Addressing, WS-AtomicTransaction and trying to patch several frameworks, he basically gave up and dumped the whole BPM tool.

And the next customer kicked out a vendor during a pitch early, because they wanted extra money for providing a Java API.

All these examples have one thing in common: The tool made the developers’ lives harder instead of easier! They failed in reducing development time or money. They didn’t help any further in Business-IT-Alignment, since the technical models needed were much too complex to be aligned with the business process models at all. Did you see any BPEL model having anything in common with the original drawing from the business guys?

So, is BPM not working? Will Business-IT-Alignment stay a myth? Of course not! However, we have to rethink the way we do BPM projects. We did that over the last years and found a way how BPM can work in real life projects.

In short, it is much more about collaboration and respect for different roles, and letting people work the way they want. Hence, it is less tool-centric, because even if we need tooling, it has to adapt to the way we work. We have to stop the situations where tools force us work in a vendor prescribed fashion. Different roles will use different tools, so there will not be the one and only tool. Even if that seems obvious, many tools still try to do exactly the opposite.

We developed a methodology which uses BPMN (Business Process Model and Notation) to gain working Business-IT-Alignment. It is about using the process model as a center for collaboration where we discuss and link requirements, business rules or other artifacts, visualize development statuses, specify business driven test scenarios, and much more. It is about modeling not only the executable process, but also about the organizational aspects in surrounding pools to align the business and IT view.

Therefore, BPMN offers a great possibility: pools. This makes it possible to model "business only" aspects and technical aspects in one model and set the right relationship between them. We described this in more detail in our book and committed an example to the official BPMN example document. To get an example of what I am saying, you can have a look at a blog entry from my colleague as well. Overall, we see all of that as the real value of BPM, not the business guys’ "drawing" executable processes.

We executed that method with some customers already, some even using Microsoft Visio. Over the past months, however, we also developed tooling to support this approach with a couple of pilot customers, most importantly one big project at a large German telecommunications company. Currently we release all of that open source as camunda fox, the first version of which will soon be released to the public. We want to share our experience, gain new ideas by discussions, and help BPMN to get adopted the right way. Then it can deliver real value to everybody, instead of only to the bank accounts of tool vendors ;-) And hopefully, we can skip the “trough of disillusionment” seen in the Gartner hype cycle, especially for BPMN 2.0.

Let’s get more concrete and have a look at the telecommunication customer project I mentioned. The environment we faced here was basically Java EE using a lot of EJB, some JMS, and a limited amount of JBoss ESB services. Processes were not yet consistently documented, the business department mostly used Event-driven-Process-Chains (EPC), and the IT folks used everything from UML to Power Point. The goal was to introduce BPMN as a modeling notation for Business and IT and as a bridge to keep the technical and business processes of the model in sync.

A goal we achieved in the end! But step by step…

Technically a “pimped” JBoss SOA-Platform is in place, meaning executable processes are deployed to JBoss jBPM 3.2, the well known Open Source Process Engine. The decision for jBPM was made after some unsuccessful attempts with BPEL, which just did not fit the technical environment. The main reasons were basically that services were not available as Web Services, the tooling was not mature enough, it was too expensive and the know-how was lacking. Hence, jBPM was a good choice to get existing Java developers on board with minimal disruption of the development process. I want to point out one important aspect here: Open Source Process Engines like JBoss jBPM or lately Activiti are more like a framework or library to the developer than a full blown product suite. They can be easily customized and integrated in your own architecture. They allow unit testing and are easily understandable.

Remember: We want to give the different roles their necessary tooling. Developers are happy with Java, Eclipse, JUnit, Subversion, Maven and so on. Hence they should be able to keep using these tools!

The business analysts have a commercial Signavio edition in place, a tool the business analysts are happy with. The developers are not forced to use it as well because it is open enough to access the stored models on different ways.

The question is: "How can we have the models in sync if we have different tools which use different repositories?" The solution is simple: we created a glue "layer" between the different tools and repositories. This glue contains of a simple web application being able to access the Signavio repository for the process models as well as the SVN for the technical artifacts. Based on artifact types, we could hook in special actions. For example to create a jBPM model from a Signavio BPMN model. This glue layer became part of camunda fox and is shown later.

It is important to keep business process models and technical process models in sync as it is the only way to keep the business process models up-to-date. This is exceedingly important not only for documentation purposes but also to report KPI’s, or the like, on the level of the business process models. However, we could achieve that with the relatively easy glue layer, without the need of a high-sophisticated zero-coding tool.

To see the glue layer in action, I will show some example with screenshots. Let us imagine a business analyst creates a business process model and finishes his first iteration. He wants to hand it over to the IT team so this process can be implemented. He notifies the technical project lead that he can start on the work, which is easily accomplished by sending a link via E-mail. The project lead now has an action available for the BPMN model to create a development project in SVN. This is done by using a project template, which is filled with a jBPM process definition created out of the BPMN model. The latter is done by a special mapping, which we implemented. The provided basic mapping can be extended company-wide, department-wide or even project-specific conventions and patterns. You can find some reasoning why a mapping may make sense, even with BPMN 2.0, in my blog.

Than the developers working on the process, doing all the dirty technical details, adding so called ActionHandlers to the process responsible to execute Java code during the process flow. That feels basically like any other Java-Development project.

The process doesn’t have to be deployed onto the SOA Platform to be tested, which allows normal JUnit testing, continuous integration and the like. Basically, it allows for typical Java and jBPM development :-)

You may have guessed that the connection between the Signavio and jBPM models is saved in the background. So when the developer commits, we can see that in the camunda fox web application. Not every commit means that the changes should be merged back into the BPMN model, so the developer sends an email to or schedules a physical meeting with the business analyst. They can use the fox GUI to see an easy diff and copy the changes back into the Signavio repository.

We were pretty successful with that approach. At the end of our project, the BPMN diagram used for documentation and the executable jBPM process were in sync. And the people liked it for different reasons: BPMN has acceptance in the business departments, and an up-to-date process documentation was truly appreciated. The lightweight Open Source Process Engine was handy in development and proved to be intuitive for Java developers, which I think is one of the most important points. The glue in between added the missing link between the two. In the meantime, we tried this approach with another customer in a different project, and it works there pretty well too. But keep in mind that this approach is just an example; if you want to work differently, work differently and adapt the tools to suit your needs!

There is another "low hanging fruit" that we try to harvest next. Because we have a link between the different models and that glue layer, we can have a central entry point that provides an overview of the process. We can see which versions exist, which version is released on the business side, which BPMN version is deployed as a jBPM process on the engine, what different versions run in parallel on the engine, and so on.

Even on the engine, there might be different versions running in parallel. The links can also be used to show KPIs measured in the jBPM engine on a BPMN level. The following picture shows an example of what this may look like.

To be honest, what we have done here is just the tip of the iceberg! Using this as a starting point, there are a lot of possibilities to extend that glue layer. One interesting direction is to support distributed teams such as off-shoring projects. There you need more collaboration features. Features like discussing the process model and archiving linked email discussions. This is only one step away from the idea of having generic access to repositories and artifacts and be able to tag and link them to the process model or with each other. This allows traceability from a project order written in Word and saved on a Network-Drive to the BPMN diagram to the technical model. We are currently introducing some kind of tags and virtual folders to improve the user experience as well as the overview. For another impression on this you can check the Activiti Cycle Vision, which heads in a similar direction on this edge.

An even more interesting direction that we want to tackle in an upcoming customer project is to add better support for process testing, as I described in my blog entry. Have I mentioned that we are working on a prototype to be able to dynamically show the dataflow of processed data used or produced in Java classes which are attached to the processes on a BPMN level?

This gets especially handy if you implement highly-automated processes "just" calling services, where you invest the most time in the question: "Which data do I have where, and who created it?" Last but not least, I see great potential in combining processes and rules. Not new? The open glue layer allows you to easily create rules in a decision table on the business level and link them to the BPMN process model. In the executable process, this boils down to a small portion of Java code, calling e.g. JBoss Drools, a fantastic Open Source Rule Engine. While some big vendors already offer something similar, our approach will bring the feature to the Open Source world as well.

I hope I have given you an idea of how we got the business process model and the technical process model in sync. What I want to sketch is that there is a pragmatic way of building some easy tools to support the approach you need. Furthermore, since it is Open Source and extensible, you can take it and implement your own ideas, extending the tooling to meet your individual needs. We have already had very good experiences with the sketched approach, and we will continue to investigate and develop camunda fox, working hand in hand with our customers and through ongoing pilot projects.

Of course, camunda fox isn’t a silver bullet as well ;-) But if your project involves (Java) development, I think it is worth your time to experiment with camunda fox and give us feedback! We are still learning a lot every day, and I am curious to see where we end up. I strongly believe, however, that all the effort invested will lead to a radically different approach in developing process-centric applications. And the good news: I think that’s good for everybody :-)