BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles The Seven Fallacies of Business Process Execution

The Seven Fallacies of Business Process Execution

This item in japanese

After 8+ years of intense research, the software industry and its customers are hitting a wall. The vision defined by BPM startups in the dotcom era has not materialized yet: we are still far from having the ability to use the business process models designed by business analysts to create complete executable solutions (even with minimal interventions from developers). The need for process driven application models is real: Business Process Improvement initiatives are humming and running everywhere in G2000 companies, but despite such a strong need to continuously improve processes, the BPM market still remains marginal in 2007 (compared to what it could be). This contrasts sharply with the language of some vendors that had been prompt to portray themselves, in 2000,  as the next Oracle for the Business Process Management System (BPMS) space... 

So what happened? It is actually very easy to understand. It is the usual "I'll sell you what you want to buy" story. In these types of situations a series of misunderstandings usually arises which leads to suboptimal solutions. If you add into the mix that most product managers, architects and developers never ever talked to a business analyst, let alone try to design a business process by themselves beyond a few boxes and arrows, the current situation should come as no surprise to anybody.

Last week, Bruce Silver asked the critical question of "Roundtripping revisited". Bruce complains that there is a strong mismatch between the two key standards of BPM: BPMN (the Business Process Modeling Notation) and BPEL (the Business Process Execution Language). He pointed out the outstanding work of a team of researchers (Ouyiang, Dumas, van der Aalst and ter Hofstede) that set out to create a BPMN to BPEL compiler since it is often argued to be the missing link in current BPMS architectures. They have made great progress to solve this problem, but their work is still incomplete. He also argued that we should give up on BPEL altogether and focus on what appears to be the successful path: creating an executable BPMN standard layered underneath the notation.

I have been working at this problem since 1997 and I have written in 2002, two articles(1,2) which have been both referenced in the OMG BPMN 1.0 specification. I would like to reiterate the arguments that I developed in these articles, perhaps more clearly, with a different example. My goal here is to explore the misunderstandings on which the current architecture of BPMSs is based and offer a new architecture blueprint on which a new class of Business Process Management Systems could be built.

Fallacy #1: Business analysts model their processes from a systems' point of view

If you talk to practitioners they will tell you they model processes from the user point of view and not from an execution point of view or a systems' point of view. It means that their process model instructs the user what to do, they never model the responses of the systems to the user input. There is a good reason for that: business continuity. If all systems fail, users need to know what to do for the business to continue operating. It is also the way business analysts think and how they define and get their metrics from processes, this user view is very important to the business because it directly relates to the workflow of activities that creates value. Business analysts never think in terms of system boundaries, execution, messages or business objects (but developers do). At most, the business analyst's understanding of a system is a screen which really amount to an electronic version of a paper form (to view or enter information). 

Fallacy #2: Business users can easily learn BPMN and use all its features.

BPMN is a 300+ pages specification. It is elusive to think that even a fraction of your business analysts will be able to master all these concepts. Michael zur Muehlen has run a survey of the most used constructs in BPMN (see slide 24) and his conclusion was that about 25 constructs are routinely used. Personally, I have created a tutorial for business analysts based on 10 key concepts and even by pairing down BPMN it was hard to convince the Lean Six Sigma Black Belts I worked with to adopt BPMN.

Bruce Silver comments from experience (as he teaches BPMN classes):

BPMN has a lot of attributes put in there just for BPEL generation, and these are generally ignored.

Fallacy #3: Business analysts should be able to create executable solutions from process models

I am not saying BPMS vendors are disingenuous in trying to sell you a BPMS under the argument that it actually does that. BPM started with good intentions: the vision of better Business/IT alignment, faster development cycles... The idea emerged that the business could actually produce models that could be turned into executable code. No wrong doing there, this is in the same line as CASE tools, MDA, MDD, DSL... This vision spoke to our dearest dreams: fast, easy, cheap. Each time I hear a vendor's spill on this topic I think of John Lennon's song Imagine (i.e. I want to live in this world, but it is not going to happen in my lifetime). Vendors felt there was a real (and huge) market based on a solid idea and when you combine that with the almost infinite pool of money that flew from the VCs, well you get what we have today. Some vendors succeeded better than others at delivering a fraction of that vision, but we have to admit that the vision is not there. Nobody can claim they have delivered a general purpose engine that business analysts can use (even with minimal intervention from IT) to create a solution from process models. Big projects fails and BPMS usage is marginalized and bring little benefits to an organization.

The joke I often tell people that want their business users to "craft" solutions is: the good news is that you just added 2000 developers to your organization, well the bad news is that you just added 2000 developers. You want your user to be able to personalize solutions, not to build or even customize them. Note, that in some well constrained cases, it is ok to let business users customize some of the business logic (such as rules).

Fallacy #4: If we add a magical BPMS that create solutions directly from business analysts inputs we would not need to develop any of integration with existing systems nor to change existing systems of record nor to do any QA.

Stated that way, I hope that by now everyone agrees that we will not see such a magical BPMS on the market for at least another 10 years. And yes vendors have given up completely on taking developers out of the loop. Bruce notes however:

a host of smaller companies began to demonstrate success both with BPM buyers and industry analysts by ignoring BPEL altogether. Vendors like Lombardi, Appian, and Savvion, focused on human-centric processes more than integration, led the way with a new style of BPMS in which executable design is layered directly on top of the process model, in the form of implementation properties of BPMN activities.

The tooling itself encouraged business-IT collaboration throughout the implementation cycle, and fit well in agile iterative methodologies that significantly shortened the cycle time from model to deployed solution.

Marlon Dumas who responded to Bruce agrees with me:

You won’t remove the developer from the BPM lifecycle, simply because no business analyst will ever be willing to write something that resembles an XPath expression, or any other expression language.

I would argue, as I said earlier, that these vendors have experienced limited success. As Bruce points out they focus on human-centric processes, which I agree for the most part fit well the centralized view of a business process engine developed by these vendors, especially when limited customization of and integration with existing systems is needed. .

Fallacy #5: Business Process Execution must be centralized

Let's spend some time on this one. Bruce explains that he is confronted to a new problem:

In fact, more often than not, if [his BPMN users] have already made their BPM runtime decision, it is BPEL. It’s a standard, a commodity, available open source.  It’s what IBM and Oracle use in their BPM runtime.  So there are compelling factors in BPEL’s favor.  But standardizing on both BPMN and BPEL?  No, of course it’s not logical. 

Having been in the roundtripping-is-dead camp for about a year, I now find myself having to confront this issue once again.  In my BPMN training, for example, students want to know what strategies or patterns should they use in their BPMN diagrams that will fit well with their expected BPEL implementations.  It’s not something I expected to think about when I started.

A BPMN / BPEL round-trip has been the holy grail of this industry. This was the vision initially proposed by BPMI.org the founding organization of BPML and BPMN. What happened there? How could a few companies have created a successful market for human-centric processes without the need for an intermediate orchestration language when they added some execution semantics to BPMN? Others suggest that the problem comes from the fact that we have not yet found the right coordination language. Arzul  Hasni for instance suggests that GRAFCET could be a better candidate than BPEL to achieve this round-trip. GRAFCET is a programming language dedicated to industrial automata (Arzul gives details in his post). In essence it is fairly close to BPEL.

Ouyiang, Dumas, van der Aalst and ter Hofstede did a remarkable job at creating the BPMN/BPEL mapping. For those of you who, like me, have forgotten most of their college math, I published these UML diagrams for BPMN and BPEL, they may help you understand the divergence of the semantics (i.e. the things you can express in one and the other) between the two specifications. The conclusion from this group of researcher is pretty clear:

A possible avenue for future work is to extend the proposed techniques to cover a larger subset of BPMN models, e.g. models involving exception handling and other advanced constructs such as OR-joins. Unfortunately, many advanced constructs of BPMN are under-specified and are still being refined by the relevant standardization body.

This concept of a centralized process engine is not new. This is the foundation behind 99.99% of the work that has been done in this space since the early 90s. This focus on centralized architectures can be best understood by this excellent presentation from Keith Swenson, VP of R&D at Fujitsu Computer Systems (who is very invested in XPDL an interchange format for BPMN).

Unfortunately, this view is completely flawed and I would like to spend some time explaining why. With this kind of thinking we are simply ignoring the very nature of business processes: enabling an organization to add value by transforming resources. Processes such as Source-to-make, Quote-to-cash... all move "things" along a work flow of activities that ultimately (and hopefully) add value to the resources being transformed and consumed. The information systems are simply here to advance, capture and report the state of these resources and activities. Yes, you can take any business object that describe a physical concept: Purchase Order, Invoice, Inventory item, Employee, Customer... they all have a lifecycle (that can be described by a state machine - see figure 2.). 

I would like to take the example of a Job Application business process (this is the Candidate-to-Employee process) that takes a candidate application and processes it to the point where the candidate can either be hired or his application can be rejected.

Here is a typical Job Application information model.

Figure 1. The Job Application Data Model

This job application has a lifecycle (please note that the Job Application data model -the content- is independent of its lifecycle and vice versa):


Figure 2. The Job Application Lifecycle

The Job Application lifecycle itself is independent of any Candidate-to-Employee business process. This is a piece of business logic that changes rarely even though the processes that interact with it, might change often. A company could also have several of these processes for the same lifecycle: for instance, one for VP positions, one for managers and one for all the other employees. Another case would be because of regulations, some processes may involve additional activities (background check...). These process variants are extremely common. However, for the most part a job application is a job application and even though there could also be some job application lifecycle variants they are for the most part decoupled from their process variants.

The question now is how would you go about implementing this Job Application Lifecycle component? The way I would do it is by creating a service that implements all the actions that will result in a state transition:


Figure 3. The Job Application Service

All these service operations will in effect execute some business logic that will result in a state transition.  What's the best language to implement this service? Java/C#? BPEL? GRAFCET?

My preference is a message oriented orchestration language like BPEL because these resource lifecycles are long running (days, weeks, months, years). To illustrate that point, let's take the example of a customer resource: as a customer, I just canceled a 12 years relationship with a credit card company this week, (which cause the lifecycle of my customer instance to transition to its final state) because I had to pay some extra fees due to what I felt was a broken billing process... Yes processes do matter, and they could have added an activity to their process without ever changing the Bill lifecycle that would have kept me happy, but they didn't, they chose to maximize fees instead. BPEL is an ideal implementation language for such long running lifecycle (not processes) because it understands messages (receive, send, invoke), message correlations, and it can deal with parallel flows (yes a resource can have composite states). In addition, BPEL engines have been designed to automatically handle dehydration/hydration of process instances which is one less (painful) thing to implement.

The BPEL implementation would look like this (using a vendor neutral BPEL notation):

Figure 4. The Implementation of the Job Application Service

I know a lot of people will tell me that it is a process, but it is not. It is a service implementing the lifecycle of a Job Application independent of the processes and activities that may advance the state of the job application. A process is the set of activities that advance its state. Resource Lifecycles and processes are decoupled, I don't think anyone can argue with that, yet everyone is trying to model and implement processes without a clear understanding of the resource lifecycles, they are more or less "built-in" the process model.

So the choice that most people have made to standardize on a BPEL engine is the right choice ... by far. Note that because of SCA, your favorite programming language can easily be extended to incorporate BPEL semantics. In the past I would have favored BPEL-J over BPEL but today if you need to express some business logic in a traditional language, SCA makes it really simple to leverage orchestration capabilities in your favorite language (Java, C++, COBOL, PHP, ABAP...).

There is such a strong relationship between resource lifecycles and orchestration languages that the leading orchestration engines offer a state machine paradigm as a way to create you orchestration definition. This is the case of IBM Process Server, and Microsoft Workflow Foundation. (I apologize if I forgot some, please let know if you know of others).

Please note that so far I am suggesting to use an orchestration engine to implement the services that manage the lifecycle of resources, I have not talked yet about business processes or business process engines. 

Before we start looking at the relationship between a lifecycle and process, let's emphasize that a lifecycle is a very intuitive concept. Most business analysts could easily describe these lifecycles readily (say using a UML notation). I would argue that almost anyone in an organization can understand these lifecycles, whatever their roles are. However, I would also argue that on the opposite end of the spectrum almost no-one would be capable of designing (as in graphically designing using BPMN) a business process that would comply with the lifecycles of all the resources involved. Assuming you created such a model, let's say that you now create a process variant. How would you be guaranteed that the resource lifecycle was not impacted? How much QA do you need to do to verify that?

Process and Resource Lifecycles can only be reconciled during the process implementation and possibly "bending" the process to make sure it complies with the lifecycle. This activity can only be performed by a developer mapping carefully the requirements of business analysts expressed in BPMN and reusing the enterprise class services that manage the lifecycle of the core resources of his or her organization.

Now, let's look at how a business analyst would create a Job Application business process definition using BPMN:

Figure 5. The Job Application Process (The groups in Blue represent Human Task boundaries)

First, BPMN does not have the notion of "resource" and a fortiori "lifecycle", at best someone could annotate its BPMN definition with the expected states at a given point in the process (as shown above). This is perfectly fine, this is how BPMN should be. Second, the business analyst is totally unaware of the operations that will be invoked on the Job Application service to advance its state. They belong to the systems view. Expecting the business analyst to add "invokes" activity in between the activities he or she describes as user activities is simply dead wrong. Unfortunately the relationship that people set out to establish between BPMN and BPEL was the wrong one and they ended up adding the core BPEL operation semantics of Send, Receive and Invoke in the process notation. This is totally artificial and should be never used, unless the message being received or sent is a business business message not as an operation being invoked (such as a Job Application arriving on the desk of a recruiter).

How does a business process gets implemented? A business process execution environment is an assembly of services (figure 6) interacting with each other (not a centrally orchestrated set of services). It is the interactions of the orchestrations implementing the resource's lifecycles as well as the performance of human tasks, the events and simple service invocations that advance the process.


Figure 6. The Job Application Process Implementation

The great news is that we already have all the technologies necessary to achieve this vision including an assembly technology: the Service Component Architecture. Everything you see on this picture can be achieved with a combination of SCA 1.0, BPEL 2.0, Web Services (XSD and WSDL 1.1 -because of BPEL 2.0), BPEL4People 1.0 and Human Tasks 1.0.

In this BPMS architecture blueprint, the statement made earlier by Bruce:

With BPEL you don’t have the freedom to ignore elements you don’t support.  BPEL is BPEL and you have to support everything in the spec.  The rest are called proprietary extensions.  They live in their own namespaces, and a valid criticism of BPEL 1.1 is that real processes need too many of them.  It’s a bit better in BPEL 2.0, but human tasks, subprocesses, and other basics still require extensions in 2.0, such as the nearly mythical BPEL4People.

does not apply anymore. WS-HumanTask and BPEL4People belong to the task container and are indeed separate from BPEL itself. Now you can argue whether BPEL needs "subprocesses", but I would say that as an implementation language of Resource Lifecycle Services it is not critical: very few elements of a state machine are that reusable, they belong intimately to their resources.

At this point -and unfortunately- Microsoft is not participating in SCA or BPEL4People so you cannot use Workflow Foundation as an alternative to a BPEL engine even though it would do the job perfectly. You can however use WCF as a service container implementing services that can be invoked from SCA and your favorite BPEL engine. Microsoft itself does not have an assembly mechanism so you cannot even implement this architecture blueprint in .Net. On the open source side, you have most of the components (SCA, BPEL, and Service container) but a BPEL4People container is missing. It is not critical, basic human task containers are actually not too hard to build (but not to the level of BPEL4People and WS-HumanTask).

To understand the role of a developer in this new architecture, let's focus on the "Schedule Interview" activity of the process model (Figure 5). As you can see this activity is featured in the process model (and it makes sense because, if the job application system is down, this is what a user would have to do), but as an optimization it was decided with the business that the scheduling task would be automated on top of exchange server for instance. The Job Application lifecycle provides the hooks (i.e. requires) that an interview be scheduled after the candidate's application is retained. Note that the Job Application service does not know how this is implemented. It could have well been a human task too. At this point of my understanding, it is simply impossible to resolve this kind of design decision automatically. This is why process models must be completely separate from any execution semantics. Another design decision that would not impact the process definition is the fact that the candidate application could happen in a different human task container. We could very well "assemble" this process with the candidate application taking place in a popular career site. Once the application is approved for interview, an activity would send an email to the candidate to point him or her to the process tasks (review offer, enter employment information). I bet you can't do that (easily) with your current BPMS architecture.

As a side note, you can see now that a task engine is not really a sub-component of a business process engine. Of course, this is how BPMSs are designed today, but in reality it is not, it is an independent component of the architecture managing human tasks (figure 6). These human tasks are naturally always related to one or more business processes, but they have a lifecycle on their own and interact directly with the resource lifecycle services. As Dominique Vauquier[1] puts it in his article: "Human tasks are grafted on the resource lifecycle". In addition, as we have seen in the previous paragraph it is critical to enable a "business process" to interact with several task containers.

I did not describe here the role of rules or Master Data Management here (apologies, James) but they do play a crucial role and require a specialized service containers a.k.a. BRMS (Figure 6). The question that Michael zur Muehlen or Mark Proctor ask becomes totally irrelevant because SCA makes it irrelevant (from a runtime perspective). SCA will let you choose the most appropriate invocation mechanism of a decision service (running in process with your BPEL engine if it is technically possible). SCA offers a large degree of decoupling between the elements of this architecture allowing them to be reused in different processes while choosing the best runtime configuration possible for each process.

I did not speak of the role of B2B either, I speak a lot more about it in my two original articles (1,2). This architecture blueprint supports B2B by enabling the definition of arbitrary boundaries within the assembly. For instance I can "assemble the two views of a purchase order lifecycle (buyer and seller). This is a tremendous advantage. Traditional "centralized" execution models impose an artificial discontinuity at B2B boundaries and force two different execution models: a centralized orchestration on each side and an assembly in the middle. In some ways my proposal is simply based on the original B2B process definition model of OASIS ebXML Business Process but applied at the resource level, not just at the business partner level. This is why the execution models are continuous both inside an organization and at the periphery as it interacts with its business partners.

Fallacy #6: Business Process Execution semantics can be derived easily from existing programming concepts

Pretty much everyone I encountered in the "execution" standard working groups (such as BPML, BPEL, WS-CDL) were not practitioners (that includes me). They were developers and architects. They often focused on complex mathematical theories (such as the Pi-Calculus) without ever validating if these theory's semantics would actually be enough to support a business process execution. Typically, these technical committees would focus on 3 to 5 use cases to write their requirements. These use cases would be often trivial, and would rarely reflect the "real-world" complexity of business processes.

Business Process Execution semantics are difficult to conceptualize. It is actually so difficult that most executable processes are still painfully hard-coded in our solutions, one line at a time. If there was a better way I am sure everyone would embrace it. I was encouraged to read the comments from the "Why Java Developers Hate BPM?" discussion. Not one comment complained about the validity of the abstraction. Even code Kahunas, such as JBoss's Chief Architect, Bill Burke (with whom I worked briefly as we were building together a human task container before he joined JBoss) comments:

I thought the same of BPM. That it was nothing more than XML scripting and the dumbing down of developers. Until I actually started looking into BPM frameworks ... I didn't see the value add these frameworks had to offer. When I started thinking [about them] as a reliable and fault tolerant state machine I really started to see the potential for BPM frameworks. Then when you start combining the use of transaction management and compensations with your business processes, you have some real nice abstractions to work with as you develop your applications.

Based on what I explained in the previous section, his and other's statements go in the right direction. Developers now see the difficulty in having to code state machines over and over and how a generic engine could ease their job (in most cases).

Fallacy #7: Bruce Silver concludes his post by saying that "the collaborative implementation paradigm, in which executable design is layered on top of the BPMN model, is the way to go."

Bruce believes that a business process model implementation should be driven from the business process model expressed in BPMN and successively add annotations (collaboratively with developers) to achieve an executable process.

Unfortunately this vision does not take into account the reality of business processes (as a work flow of activities that advance the state of resources) and I hope I convinced you that this statement, even though, conceptually, a valid endeavor, could not be more wrong because we cannot model workflow of activities and resource lifecycle together (at least in the current state of my knowledge). I can foresee for quite some time that developers will translate the process definitions produced by business analysts into assemblies of human tasks, resource lifecycle services and services (including decisions and MDM services).

Now, this new architecture blueprint does not mean that the investment you made in your favorite BPMS is lost. You will need however to add a Composite Service Container (such as a BPEL container) and an assembly container (SCA) and use your BPMS mostly as a human task container (which they actually are, for the most part, anyways). A human task container is a noble and important component of the architecture. The current BPMS's task containers are very sophisticated and would be difficult to build yourself, so it was money well spent. I don't want to undermine the role of this container at all. I am actually expecting than within 2 years all the BPMS vendors will have adopted the vision presented in this article and transformed their suite to work within SCA assemblies and BPEL containers based on this blueprint.

I also argue that, at the end of the implementation, it is possible to re-construnct an "as-is" view of the operating process automatically. I have not proved that, this could become a research topic.  

Conclusion

After so many years searching for the BPM Magic Bullet, the software industry is facing a wall. This wall can easily be overcome with a paradigm shift and a new factoring of the business logic based on Resource Lifecycles. If we take the wrong turn today and still believe that in these 7 fallacies, we are running the risk of having to throw all these products and standards away for lack of ROI and return to coding everything by hand. If we, however, take the very same technology we have today and use it differently, we can deliver a vision that is very compelling to both the business and IT. I would not call that vision BPM per se, it is larger than BPM, I would rather call it "Composite Applications" or more exactly "Composite Solutions".

The Composite Solution vision speaks directly to what the business needs from IT:

  1. Build solutions rapidly with projects as small as possible (rely on many iterations)
  2. Change solutions rapidly and support an iterative lean six sigma approach
  3. Be able to visualize the business design in operation at the present time without complex “current-state” projects
  4. Be able to gain operational intelligence from the current business design without complex measurement projects

I argue that the capability of "Being able to build / change the solution directly by creating / changing the business design" (no matter how desirable it appears to be) is antagonistic to these four requirements. The reason is because it leads to simplistic and rigid task centric application models (as we can see in BPMSs today). These application models cannot meet the needs of the business and typically result in increased project cost because when real solutions need to be developed, they require a lot of custom developments “around” the BPMS application model. To compound the problem, these suites, as pointed out by the "Why Java Developers Hate BPM?" discussion, do not offer yet a robust development environment for this custom code and suitable for large projects.

I argue that the vision moving forward is Composite Software (based on two composition models: assembly (SCA) and orchestration (BPEL) -ok, choreography is coming down the road - of course - but that I will explain this in another article. The technology to develop this blueprint is available today. In addition, BPEL and BPMN, as they are defined today, work. If something needs to be changed in BPMN, it should be removing all execution semantics, it should be designed to let the business analyst express himself or herself. If you want some more details about how to construct composite software using these standards and this architecture blueprint, this mini-book was published on InfoQ last week.

The architecture of Composite Solution Platforms, as described in this paper, also offers a cleaner interface between SOA and BPM. It gives SOA the opportunity to build truly reusable services: the Resource Lifecycle Services which can be reused across process domains and process variants. Because these Resource Lifecycle Services become reusable across processes, it also means that the implementation of any given process becomes that much cheaper, faster and easier. The implementation of Resource Lifecycle Services are the "code" within the process. Thinking that a business analyst (or anyone else) would have the knowledge to code and recode these lifecycles in a graphical notation amidst the process definition is simply pushing BPM in the wrong direction.

This blueprint, as a composite solution platform, has already an Enterprise Method that can support it: Praxeme.  The Praxeme Institute is translating their artifacts in English and making great progress towards this goal.

Now, I do share some of the concerns from Bruce and Marlon about involving developers in the current technologies (SCA, BPEL...), this is why I have started an initiative called wsper. This initiative offers a abstract programming environment to simplify the work of developers and architects during the lifecycle implementation and process assembly. It also helps construct a Composite Solution Platform from heterogeneous components because it isolates the business logic implementation from these components (and their future evolutions). It also isolates the business logic from the evolution of standards.

I want to extend many thanks to Sandy Kemsley for providing so many useful links and comments.

[1] This article complements Dominique Vauquier's article ("The 6 Fallacies of Business Process Improvement"). Here we focused on business process modeling as it translates into execution. Dominique's article explore how Business Process Modeling relates to Business Process Improvement projects. I translated Dominique's article from French (it has been accepted for publication on BPTrends.com in January 2008). For those of you who read French, the article can be found on page 39 of the "Guide of the Pragmatic Aspect" of the Praxeme enterprise method.

Rate this Article

Adoption
Style

BT