"SOA Governance" Revitalized
Reviving an often-discussed but poorly understood topic
The term “SOA Governance” has been used in the industry for years already, but it is still considered an arcane topic. Anyone who reads this article should be able to understand the following things:
- Why are people pursuing SOA, isn’t it dead?
- What is the relationship of SOA Governance to SOA itself?
- What is SOA Governance?
- How does it differ from Management?
- How does SOA differ from Integration?
While being able to think and speak clearly about these topics may not win you as many brownie points as it would during the time when this was a “hot topic” for Enterprise IT, it will help you understand why SOA and SOA Governance continue to be significant issues for the Enterprise despite the ups and downs of market hype cycles.
Why are we still talking about SOA?
Anne Thomas Manes posted her famous “SOA is Dead” post at the beginning of 2009. Since then, some of the “fizz” has gone out of SOA. But by making two irrefutable statements, we can clarify why, in the words of Gartner Analyst Paolo Malinverno “SOA is Inevitable”.
- Enterprise IT continues to suffer from bad architecture
- Since SOA was “declared dead”, no viable alternative solution for this problem has been proposed.
In the spirit of InfoQ, I challenge anyone to either
- Propose a viable alternative that addresses the problem of “Enterprise Architecture Syndrome” as defined below. Or
- Show that a viable alternative has already been proposed
- Make a case that “Enterprise Architecture Syndrome” is not a significant problem.
Feel free to attack the premise that Enterprise Architecture is in bad shape as well. Please keep in mind that:
- This is an Architectural problem and requires an architecture solution (so programming styles like Events and Objects solve problems at a different level)
- Cloud Computing is not an Architecture, it is a delivery mechanism
- Business Process Management without SOA does not reverse “Enterprise Architecture Syndrome”
One of the challenges in maintaining a paradigm such as “Service Oriented Architecture” is that during the “peak of inflated expectations” in its hype cycle it became a “big tent” under which many many software companies huddled. Everything was marketed as “SOA” in the hopes that it would sell better. Of course the billions of dollars spent helped to blow up the hype around SOA to epic proportions.
Throw out the SOA Baby with the SOA Bathwater
One of the difficulties associated with SOA is the length of time it has taken for the industry, products and implementation methodologies to mature. Because of this achingly long time span, contemporary SOA bears little resemblance to the fanciful ideas of a few years ago.
So the question naturally arises—has the term outlived its usefulness? Wouldn’t the community be better off throwing out both the baby and the bathwater and just moving on to the next concept? This seems a very legitimate approach, but since renaming my blog http://soacenter.com to http://whatevercenter.com in response to Anne’s “SOA is Dead” post, there has not been a viable alternative to solving the disease that I call “Enterprise Architecture Syndrome.”
The Disease: Enterprise Architecture Syndrome
Most large organizations suffer from a degraded architecture that I refer to as “Architecture Syndrome”. Much like the human disease called “Metabolic Syndrome”; this disease has many risk factors including a heterogeneous and aging technology infrastructure, a company that has grown through acquisitions or geographic expansion, an industry undergoing consolidation, a stiff regulatory environment as well as the commoditization of platform software such as ERP systems.
Symptoms include an inflexible IT System (including hardware, software, integration and IT processes) that makes any changes into a nightmare of schedule slips and weeks becoming months. Diagnosis can include the creation of insane-looking architecture diagrams with hundreds of different product names and thousands of object models, data tables, events, processes, organizational charts, components, lines, integrations, connections, network topologies, standards, applications, schemas, documents and other abstractions.
We best capture the degraded state of Enterprise IT Architecture by using the word “Entropy” from the physics of thermodynamics… defined as the amount of energy in a system that is unavailable for work.
There are many ways to measure the energy in Enterprise IT that is “unavailable for work”, but you can really look at it as time and budget wasted. Wasted on throwaway systems, redundant systems, wasted on useless meetings (productive meetings are ok), maintenance and other abuses. Looking at the percentage of the IT budget that goes to move the organization forward, it’s clear that for most large organizations the majority of the IT budget goes towards “keeping the lights on” and leaving not much room for anything else.
So this is a pretty dismal state of IT Architecture. One of the first things people notice when they decide to fix their architecture when they look at it from a global perspective is the number of redundant systems. This is typically looked at as a bunch of “Silos” and the Enterprise wide goal of reducing costs aligns with the idea that these redundancies can be “rationalized”. This cost-cutting alliance between Corporate Finance and Information Technology spawned a field called “IT Governance” and SOA Governance is their red-headed stepchild.
How did Enterprise Architecture get so degraded? People with “Metabolic Syndrome” could ask themselves the same question. Typically the answer is decades of neglect.
Perhaps a deeper question is: what caused Enterprise IT architecture to be abused and neglected?
IT Project Management
The IT organizational competencies and processes that evolved during that era involve a mentality around disposable “projects” and a short term framework around “ROI”. As long as a project could return its initial investment it was a justified expense.
There’s nothing inherently wrong with ROI. However, the way ROI is typically measured in large organizations is typically as a forecast used to justify a project. Even organizations with the discipline to measure ROI after the project gets started rarely measure it over the lifetime of the project.
Let’s face it, a “successful” IT project creates a “legacy”. Very few ROI models measure the value of something over the time span of “until the organization disappears”. These stable long-running “reusable” assets form the platform atop which the business functions.
IT goes through a Volcanic Hot Age
During the mid 1990’s the volatile fuel of Venture Capital and IPO ignited into a “Hot IT Revolution” culminating in the Y2K spending cycle.
Carrying the thermodynamic perspective further, What do we mean by “hot”?
- Larger overall IT spend
- Faster change in the technology landscape
- Less mature technologies
- Expansionistic—building out rather than cutting back
IT funding and project management disciplines were evolved for this type of environment, one in which time to market was more important than cost.
During an expansionistic phase, the ROI calclulation of “pays itself back” translates easily to “let’s do it!” But like the hangover after partying every night for a week, the longer term consequences of “let’s do it!” need to be considered. Consequences that affect not only the project but the prospects of the whole Enterprise.
IT Goes into an Ice Age
After Y2K, the IT environment began to cool off. The industry, products and marketplace began to mature. Enterprise IT began to enter a contractile period.
If you have fewer pennies to spend, what types of considerations come to mind?
A few of the trends in recessionary IT seem to make Enterprise Architecture Syndrome worse!
- IT headcount is drastically reduced: Lots of people are laid off, significant expertise about legacy systems is lost.
- Short Term Thinking prevails: new CIOs are hired who have substantial cost-cutting agendas for their “first 100 days”. Long term planning goes out the window.
- Draconian measurement systems are put in place: programmers are measured by lines of code produced, QA testers are measured by bugs fixed and everyone starts to focus more narrowly on their jobs.
- What’s best for the Enterprise is thrown under a bus: the dominant pattern becomes “whatever saves my job/group headcount”.
All of these trends cause even more friction between organizations—people feel more tightly squeezed, and when people feel pressured, their tendency to care about people outside of their group (or in worst cases, outside of themselves) decreases.
SOA is Dead, Cloud is Water Vapor and Enterprise 2.0 is a Crock.
In a period where the industry “cools down”, large trends in IT tend to be based on “coalitions” of vendors associated with themes like SOA, BPM, EDA, Enterprise 2.0, etc. Of course alongside these “cooler” times, the amount of time for technology to be adopted in the Enterprise goes up. Enterprises narrow down the list of “approved” vendors to only the larger vendors and they apply much more stringent critera for approving projects. During this type of environment, vendors huddle together for safety and these hype cycles take a longer time to develop. During this kind of elongated time frames, it become easy for critics to take “pot shots” at the coalitions, especially since most coalitions consist of a cluster of strong and weak players. Some of the weaker players huddling under the SOA banner make themes like SOA easy targets for critics. “SOA is dead”, “Cloud is Water Vapor” and “Enterprise 2.0 what a Crock” have variously been said of these three IT trends.
Why SOA makes sense even in recessionary times
And a few of the trends make the logic of Service Orientation more compelling
- “Use what you have” becomes a key. Getting the most out of the IT infrastructure you have becomes a battle cry for the Finance group. This means a “leave and layer” approach to developing new IT capabilities. “Rip and Replace” goes out of fashion. Finance will not buy IT any fancy new toys until it shows that it can play with the ones it already owns
- Consolidation becomes attractive. Any redundancy or perceived redundancy in IT systems is looked at as an opportunity for consolidation. Where there were two, now there is one. This is where the concepts of “rationalization” and “reusability” come from.
- IT becomes more centralized. Instead of having dozens of buyers, central IT buys infrastructure in bulk. Just like a recessionary family buys in bulk at CostCo, a recessionary enterprise buys IT in bulk. Do the business units get as much say about the brands it prefers and features? No, but by amassing all purchases through central IT, the costs go down. The procurement office rises to ascendency.
- Opportunity Cost becomes important: The criteria changes from “let’s do it!” to “is this the best use of our time/resources?” Opportunity cost becomes a serious issue since multiple projects compete for the same penny. “Software Portfolio Management” becomes more popular.
In geology, heat and pressure make diamonds (which we know last “forever”). But periods of prolonged cooling produce sedimentary rock—rock that is formed from layers.
But at the same time IT begins cooling off, the Business World is also cooling. Large enterprises are forced to compete in more mature markets.
Business gets Process-Oriented
The definition of a recession is multiple quarters of negative growth. So the impact to the business as a whole is either less growth or negative growth.
During the throes of such recessionary trends, one naturally sees a shift from product innovation to process innovation. After World War I, Henry Ford developed the Assembly Line that radically altered the field of Business Process Management and the Assembly Line created process flow visibility by making inefficient segments of the assembly line obvious to any observer. This innovation excelled at creating non-differentiated commodity products. Henry Ford said of his Model –T “you can have any color you like as long as it’s Black”. The industrial growth sparked by this and other such innovations launched the “Roaring Twenties” economic boom.
Across the Pacific, another stalwart in Business Process Management, Toyota uses Kaizen (continuous process improvement) to transform itself from a producer of the “Toyoda Model G Automatic Loom” into a car company.
Difficult economic times provide impetus for a significant push around “Business Process Management”. The emergence of Business Process projects on the business side has led directly to the connection between SOA and BPM. Early efforts to connect SOA and BPM through standards such as BPEL fell quite short when practitioners realized that what business people thought of as their processes were not represented by such low level programming languages. Although BPEL served its purpose as a low level orchestration flow language and a declarative way to express endpoint semantics, true business processes were not expressible in BPEL leading to the rise of such efforts as BPMN and others.
Business shifts to Niche or Vertical Markets
It’s a well known fact that as companies mature and markets mature, growth slows. So how do organizations compete in slower and more mature markets? The best contemporary description of product marketing for mature markets can be found in Geoffrey Moore’s book “Dealing with Darwin: how great companies innovate at every phase of their evolution.”
In this book Moore describes a mature market where product and service fragments into niches as a “Fractal Market”. He suggests that once a market is saturated with a single product, that product competition drives niche product development and creation of products with (for consumer) narrower demographics or (for enterprise) more verticalized solutions. The example he provides is that once every household has a “regular” telephone, the market is forced to product dozens of niche variations including car phones, cell phones, game phones, PDA’s, camera phones, etc.
An extreme form of niche marketing was documented in Chris Anderson’s book “The Long Tail”. Prior to this book, the organizational focus on the Pareto Distribution was on the top 20% of the customer base that makes up 80% of the revenue. The “head” end of the distribution curve deserved the lion’s share of attention simply because focusing on a smaller number of customers costs less, and empowering those customers gains you 80% of your revenue.
In “The Long Tail”, Chris Anderson explores the 80% of customers that typically account for only 20% of revenue. By studying internet giants such as EBay and Amazon, he explored the potential of the “long tail” and the ability for organizations to focus on serving the 80% of the customers on the “other side” of the Pareto distribution.
The inherent problem with serving such customers is being able to differentiate products at a finer level of granularity, essentially niche marketing. One solution to this problem is to develop a core set of “platform” services that can be used and recombined in billions of possible ways, and by pushing those combinations closer to the customer. This means personalization, mass customization, self service and combinatorics. Starbucks coffee, Build-a-bear workshop, Apple iPhone AppStore, FaceBook Apps, eBay Marketplace, Dell.com “custom built” computers and multi-colored iMac Computers and iPods all fall into this category.
Near-infinite consumption patterns with finite provisioning
One of the properties of fractal shapes such as Benoit Mandelbrot’s famous “Mandelbrot Set” is that it is a 2 dimensional shape with infinite surface area that bounds a finite volume. In market terms, a product that has “infinite surface area” means that every single consumer experiences the product in a unique way, but the underlying supply chain is the same. An example of this kind of product is Starbuck’s coffee. Many millions of possible drinks can be composed (e.g. Venti nonfat sugar-free hazelnut decaf latte with an extra espresso shot, served hot) but these variations do not disrupt the infrastructure requirements to deliver this product at the store. Hot water, milk and coffee beans go to the store and the biggest complexity is maintaining sugar-free hazelnut syrup on site.
In business terms, the same hot water, coffee beans and milk are “mass customized” into millions of different products. Notice the term is not “reused”.
So whether business demands process oriented consumption of IT services or other forms of mass customization such as mash-ups, “platform” services, utility services, App Stores or self service, we are talking about the emergence of complex consumption patterns for IT (Complex Demand)
Enter the Alternative IT Provider
Business has complex demands for IT. If IT is unable to supply this demand, a host of alternative sources of IT capabilities emerge. Among these include traditional outsourcing, Cloud Computing, Offshoring, and a host of emerging options for business leaders.
Although the relationship of the business to an external IT provider is usually more well defined that the relationship of in-house IT to the business, outsourcing is not a panacea.
Other than in smaller organizations, the wholesale outsourcing of all IT functions to external suppliers is not realistic. In cases where only part of the IT portfolio is outsourced, this adds additional complexity to the model. Now organizations must “organize and utilize” capabilities from on-premise and off-premise and manage new organizational agendas. Although external providers of IT may keep internal IT “in check” and force them to compete with aggressive outsiders, ultimately the wholesale outsourcing of IT puts an organization into a vulnerable state where the external provider has an excess of bargaining leverage over the Enterprise.
Complex Demand, Simple Supply
Consider the example of the electricity grid. Although the standardization of the AC power outlet (it’s irritatingly heterogeneous across countries) provides for billions of different appliance configurations that can be plugged into it, the supply of electricity is a reasonably homogeneous problem. Although energy has many sources such as fossil fuel, nuclear, solar, wind power, etc, once it has been translated into electricity it has become a commodity.
One rational business response to commodity market is monopoly. We can build an IT delivery system that has a similar complex pattern of consumption but has a large monolithic supplier. If managing a supply chain of heterogeneous IT suppliers becomes too complex and expensive, a monolithic supplier of IT becomes a reasonable alternative.
One example of this is the emergence of the acquisitive Oracle Corporation who is attempting to build a single technology stack to commoditize IT. The problem with this approach is that IT is not a commodity in the sense that electrical power is. The biggest issue is not that an ESB from Oracle is so dramatically different from an ESB from some other company—the issue is that in the provision of electricity, no supplier has extraordinary pricing leverage as a function of high switching cost. So the solution of a monolithic or monopolistic IT provider in order to simplify the complex IT supply chain is a difficult solution. Another problem with this model is that it would require an already heterogeneous IT landscape to “rip and replace” their solutions with those of the commodity provider. Although electricity is highly interchangeable, the history of IT has proven that swapping out IT providers is difficult.
Perhaps a more viable model for simplifying the supply of IT is Cloud Computing. By building very large “utility” companies for IT infrastructure, we can simplify the supply of IT services into the Enterprise. This is an excellent model, and the shift to Cloud Computing enables a huge cost benefit via commoditization and economies of scale. Very large providers of Cloud Computing will emerge and due to the disruptive pricing model, new entrants will likely surpass the old providers in this field.
Simplified delivery of IT capabilities is a wonderful cost-saving device. It’s well known that IT in the Enterprise is too complex. Unfortunately, the Enterprise is already filled with heterogeneous and complex IT supply chains and in order to keep the business running, it’s difficult (or near impossible) to switch off those systems. One future scenario might be that the inefficiency of complex Enterprise IT is so prohibitive that new business entrants fueled by “clean” utility IT providers in the cloud will outgrow and replace the old Enterprises.
This model does not speak to the logical response of Enterprises to the emerging need to support complex consumption patterns for IT Services. The correct chess move for large dinosaur enterprises is not to roll over and die in favor of small mammal enterprises. It’s certainly theoretically possible for large enterprises to “spin out” dozens of agile Cloud-powered mini-me enterprises. But radical moves such as this are typically not well rewarded at the Board and Executive Management level of most large organizations. So Large Enterprises will need to come up with a solution for IT that combines both complex supply as well as complex demand.
Don’t get me wrong, Cloud Computing is happening and it’s going to happen big. Most if not all large enterprises will have significant investments in Cloud. But most large Enterprises will be unable to rip and replace existing IT infrastructure with a monolithic and simple cloud solution. This results in a complex IT supply with complex demand problem.
Complex Demand, Complex Supply
If you consider the Starbuck’s Coffee example, Starbucks sources coffee from many sources all over the world and manages the complex relationships between thousands of suppliers to deliver a well-branded experience to the consumer. Although coffee beans are arguably a commodity like electricity, the ability for Starbucks to offer supply transparency (Shade-grown Fair Trade Ethiopian Coffee Beans) or at least the simulation of transparency to the consumer requires complex supply management practices.
So for IT, how do you take this complex legacy history, complex sources of capability and layer on a new set of capabilities that can be recombined the way Starbucks does with their coffee?
IF you’re challenged with a complex IT supply and complex IT demand situation, SOA is the only rational response. In the spirit of InfoQ, again, please feel free to challenge that statement.
If the supply of IT were just a problem of managing monopolistic vendors trying to lock you in, it would still be very challenging. As if that were not hard enough, even within Enterprise IT there is increasing heterogeneity and complexity in providing capabilities to the business.
As described in my free eBook, “SOA Adoption for Dummies”, one of the significant challenges in managing in the Enterprise is the existence of “Tribes”. A “tribe” is a self-interested organizational unit. Each tribe has its own leadership, budget, priorities, performance metrics, world view, and systems.
Tribes fall into a pattern where they behave in ways that “good for me, bad for you”.
This is not garden-variety selfishness. In this difficult economic environment, not only does management obsess about squeezing every dime of value out of every employee, but each group manager is fighting to keep all of their headcount. The combination of draconian measurements and individual contributors trying to save their jobs leaves very little room for thoughts of other groups. The metrics each group are measured by drive that group’s behavior and little else.
If you want to talk about complexity of supply, the existence of Tribalism both within and outside of IT organizations dramatically increases the complexity of supply.
The statement that SOA is the only rational response to these myriad concerns is perhaps overstating the point. Lets explore why.
At this point, the astute reader may wonder if I’m playing fast and loose with the definition of “Service Oriented Architecture”. So I would like to refer to the OASIS SOA Reference Model definition, which states: SOA is a paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains.” (Emphasis added).
How is SOA Different from “Integration”
If the aim of SOA is to “organize and utilize” capabilities, presumably through a “leave and layer” strategy, how is this fundamentally different from Enterprise Integration?
The “baggage” associated with the term “Integration” is simply that it concerns the interoperability and interconnection between systems and applications. While this is certainly a useful set of competencies for SOA, what makes SOA a different layer of abstraction is hinted at in the Reference Model Definition. The notion that the distributed capabilities may be “under the control of different ownership domains” is at the heart of SOA, and in fact poses the greatest challenge to SOA Adoption, the realization of the SOA benefits.
Of course in practice, SOA operates at a higher level of abstraction then Integration, and covers a broader set of agendas… and Integration sophisticates already understand the problem of ownership domains. But it’s the reason why we need a new word. Whether we need a new word for SOA in order to dispose of the “baggage” associated with it is subject to debate. InfoQ readers, have at it.
Reining in SOA
The idea that an Architectural paradigm can “organize and utilize” capabilities does seem like a bit of a tall order. This misunderstanding is one of the failings of SOA as realized today—that Architecture by itself does not organize and utilize capabilities. Architecture does not produce such effects; Architecture done properly only creates Architectural Blueprints.
An Architectural Blueprint is sometimes referred to as the “To-Be” or “Target State” Architecture. Without an aggressive program of implementation (which is often referred to as “SOA Adoption”), the Blueprint will remain a pipe dream.
So suggesting that SOA (which isn’t even an Architecture, it’s an Architectural Paradigm) by itself will produce positive business results is like saying that the Postmodern Architectural movement will provide housing for thousands of people. Before you realize the business benefit, you need to instantiate the architecture in the form of your blueprint. Even more to the point, you need to realize your blueprint in the form of SOA Adoption.
The SOA Baby in the Bathwater
If you follow the earlier phrase about not throwing out the SOA Baby with the SOA Bathwater, what fundamental insights do we need to preserve in order for the promised benefits of SOA to realized?
Let’s go back to the definition and the words “ownership domain”. The realization that the distributed capabilities may have different owners changes the equation from technical “reusability” into a problem of creating a framework of agreements that support sharing.
What about reuse?
The concept of “rationalizing silos” speaks to the idea that we have one service instead of two. You could look at it as that single service being “reused” two times.
This is a fairly unproductive way of looking at it.
What gets people’s blood moving these days is the idea of “good use of time/resources”. Reuse is such a dead tired word that it hardly ever gets people interested in SOA. We spoke earlier about “Entropy”—the amount of energy in a system that is unavailable for work. Another way to describe it is waste.
If you take a look at the “good use of time/resources” measurement, any project that:
- Produces a redundant function (bad use of your time)
- Will be thrown away (bad use of your time)
- That is limited to one business use case (bad use of your time)
- Produces a security risk (bad use of someone else’s time)
- Crashes or otherwise fails (bad use of someone else’s time)
It’s typical for someone who is part of a siloed organization to not care as much about other people’s time—this is a natural product of working in a large enterprise where responsibility is diffused. It becomes “somebody else’s problem”. Ultimately it results in lots of waste—code being ripped out, underused, marginalized, consolidated and otherwise wasted.
Complexity is not the Enemy, Entropy is
We spoke earlier about the complex supply and demand challenges facing IT. The complexity of Enterprise IT is a direct function of the size and longevity of the Enterprise. Although size and complexity is correlated with failure, complexity itself is not the enemy, Entropy and waste are the enemy. When we look at a system that converts energy into form, we can take inspiration from the Biosphere. The Biosphere takes in solar energy and converts it into all forms of life on earth in its tremendous complexity. Complexity in the biosphere is a measure of its strength, adaptability, scale and resilience. One could argue that Enterprise IT is a “closed system” and therefore does not have the boundless solar energy input needed to reverse local Entropy as we see in the earth’s Biosphere. But that argument doesn’t hold water—Enterprise IT continuously takes in energy in the form of IT budget expenditure. Large Enterprises simply aren’t known for “pulling the plug” on IT, and in the most extreme cases will simply outsource as much of it as possible—but energy is still being expended. Much like the Biosphere, complexity is a measure of success—it demonstrates the ability to support tremendous business variation and for an organization to attack a larger number of economic niches. Those who hate “complexity” often confuse organic complexity (which is simply a beneficial form of complexity that they just don’t understand) with overcomplexity. They may try to drive the system into a state of oversimplicity. Both overcomplexity and oversimplicity are forms of waste and therefore Entropic.
Services hide complexity
The concept of a “service” is simply the notion that consumers are shielded from the complexity of the implementation. Because of this, SOA can be seen as a way of hiding complexity. Hiding organic complexity can be useful in situations where the total system complexity makes visualizing solutions difficult for the business. However, in a business context being “shielded” from complexity essentially means abdicating control over the situation. There’s actually nothing wrong with abdicating control over a process in the business world, as long as you trust the provider or the framework of accountability. What I mean by the framework of accountability is that we “trust” that other drivers on the road will not smash into us because we “trust” that they do not want their insurance premiums to go up and the hassle of having to deal with a crash. Similarly, provider/consumer contracts can provide a framework within which consumers don’t have to explicity trust a given provider but can trust the framework of accountability.
But they require trust
Ultimately, the trust issue is the biggest stumbling block for SOA. What we’ve learned is that lack of trust in business is tremendously expensive. The speed, coordination and agility exhibited by high performing teams is a function of trust—team members pass a ball or puck to where they “know” their teammate will be—without having to look. Lack of trust means that agility grinds to a halt. Everything must be verified, double checked and legislated. There’s an old saying that if people are pulling contracts out of the filing cabinet to re-read them, it’s a sign that the business relationship is ending. Similarly, in SOA overgovernance can produce as much entropy as undergovernance. Ultimately it comes down to waste, waste is the enemy.
Writing a bit about Business Transformation here on my blog also:
My 2 cents,
Just one word, upgradeability
thanks for this great article, however, the central focus of SOA is missing: upgradeability. It is not surprising, all IT vendors have failed at providing "upgradeable" capabilities and this translated into building IT assets that themselves have been very difficult to upgrade, version, maintain... Lots of companies have actually adopted an "immutable" strategy to their SOA. What an oxymoron !
Could you imagine for a second if the physical world was built of immutable services? Could imagine each time your shipping, banking, commerce... services had to "copy paste" and keep a bunch of immutable versions?
Ludicrous, right? Yet this is all the vendors have provided, this all the pundits and analysts have recommended? SOA dies from a lack of upgradeability. The Cloud thrives from making "upgradeability" someone else's problem. If a SaaS, PaaS or IaaS vendor fails at providing a forward compatible versioning strategy, this vendor will quickly disappear.
IT started to die the day the "maintenance" business model emerged. Who in their right mind would pay 20% a year for a vendor to focus on new customer acquisition and never ever provide sustainable upgradeability paths.
Beyond, that, there is nothing specific about the "failure" of SOA. It is computer science which has failed to understand the implication of architecture. We are still building software like we were doing it in the 60s, yet somehow we keep changing everything every so often. Who in their right mind would write business logic that relies on APIs that have a lifespan of less than 24 months? How many companies have software running that is out of support? What is the cost (entropy, I think you say) of simply trying to remain within support without taking advantage of the "new" stuff.
Take any product from any vendor and calculate the cost of migrating from Vn to Vn+1, including the 20% spent. Then you understand the size of the problem. You understand why there is no other way than:
a) centralizing upgradeability (Cloud)
b) creating programming models that are independent of architecture (i.e. writing business logic that can survive the elements of architecture it relies on)
Who in their right mind would continue the course that vendors inflicted on IT for the last 20 years? IT will cease to exist unless upgradeability becomes the central focus of our entire industry. SOA was the first paradigm and set of technology that supported a forward compatible versioning approach, yet no single vendors talked about it. There is no need to "revive" SOA is you are going to keep upgradeability behind. Same approach will yield same results.
>> ... the transformation aspect is about managing both IT complexity with SOA and business
>> complexity with WOA, Web 2.0, Mash ups, Business Processes, composite apps, and various forms of
Come on Miko, can we stop this kind of talk? can we stop the sexy TLAs, pumped for a few months? can we tackle the real technical challenges of upgradeability and architecture independent programming models? Can we bring Model Driven Engineering to the masses? It is not a bunch of acronyms, no matter how sexy they look.
We don't need alternatives, we need focus.
(I am not just writing this for you, but for all vendors)
Re: Just one word, upgradeability
as usual, good insightful contribution from you! I appreciate your feedback always.
I happen to agree with you about upgrade-ability--but I see the problem as being complex from an evolutionary perspective. Youre right that vendors are to blame for many ills, but the market has created a system where risk, cost and complexity are pushed across all sorts of organizational boundaries, not just between vendor and customer. In some ways the promise of SaaS is the notion that a stable interface can be defined and that "upgrades" just flow forth in a way that is hidden from the user, save for an improvement of the customer experience.
This is a lovely notion, but it hasnt been put to the test of time... The key is the establishment of a stable interface, as this is the region that gets disrupted by fast-changing innovation. I like the trend towards IOC frameworks that enable an evolutionary pattern of dynamic package management--but see this as a half-measure.
This is why Software AG has embraced agile development methodology across all R&D and why we are rearchitecting our own products using IOC, particularly with OSGi.
As far as the "Business Transformation" buzzword, I apologize for it. I just feel that a discourse needs to emerge around improving Enterprise business performance through a combination of management and technology.
Focus is good, but too few people understand the methodology of implementation.
Re: Business Transformation
At the beginning of the project Enterprise Architects are mandated by senior management to produce SOA blueprints and governance frameworks. These mandates result in great looking diagrams and documents and some some expensive tools purchased but then they are not followed by any real adoption in practice. In the end systems are implemented and delivered in the 'old' way: huge waterfall project management, missed deadlines, buget oveflows, poor code quality, system crashes in production, disatisfied customers. The excuses I hear often are "systems complexity", "aggresive timelines", "people have not enough training", etc. Yes, systems are more complex today than 10 or 20 years ago while business evolved and expanded and user expectations are much higher. However,imho the root cause of failure is of organizational nature, in the way business and IT is structured in silos with poor communication and execution.
Re: Business Transformation
The word "SOA Governance" is itself an anathema because it smacks to people of:
* Huge and Cumbersome
* Rigid and Unchanging
* Invasive and Oppressive
This is very problematic!!! The fundamental premise of a service is the offer to hide complexity for a consumer in exchange for trust--for the purpose of producing manegability.
Trust is an absolute necessity in organization simply because no single human being is capable of understanding the complete complexity of Enterprise IT. I mean a single person can spend their whole career lifetime studying just Microsoft products (for example) and this would only be a small fraction of the whole Enterprise.
So we need to trust the experts--and also to EMPOWER them. This is the principle of kaizen which is the management principle used by Toyota to create a continuously evolving organization. Evolution is made possible through extremely short cycle times for improvement, and respect for each individual as the expert in their area of specialization. Nobody knows more than that individual about how to improve their own work environment.
SOA Governance is about directly attacking the problems of Enterprise Scale systems--that today we have a complex dance of risk, cost and complexity arbitrage between organizations (yes, including vendors JJ), individuals (including C-level executives) and individuals. Today the Enterprise is a dumping ground for complexity and risk that individuals and groups within the Enterprise manage to eschew--resulting in large catastrophic failures of implementation.
If SOA Governance itself becomes part of the problem and not part of the solution, we must dispose of it along with all the other rubbish in the Enterprise. The principles that will enable its revitalization (or ultimate demise) will be to ensure that SOA Governance policies evolve at a high rate of speed, that short cycles of optimization result and that individuals are entrusted and empowered--but within a framework that prevents abusive arbitrage of risk, cost and complexity towards individual gain at the cost of the whole.
We have been failing at this over a long time. The recent Deloitte study shows that since 1965 the Return-on-Asset of US Public Companies has declined by %75. This number is even worse for Telecommunications and Technology companies.
bit.ly/7DQEzx (link to 200 page PDF, click if you want to have it)
Reversing this miserable performance may help in the much grander objective of elevating our economies out of recession. If SOA Governance hinders this effort, then good riddance to bad rubbish, but we will still need a way to create scalable frameworks for trust across organizations within and outside of the Enterprise.
My 2 cents,
Re: Business Transformation
In Computer Science upgradeability means:
a. To replace (a software program) with a more recently released, enhanced version.
b. To replace (a hardware device) with one that provides better performance.
This is the core advantages of SOA and loosely coupled modules.
This is also the one of the WS-I goals.
We cannot achieve it if we use RPC style communication but we can achieve it very easily if we use the document style that encapsulates the messages and the data.
With Document style, a change in the structure of a message (like, adding a new attribute) does not affect the document exchange mechanisms.
On the other hand, a change in a back-end (internal) method signature has generally no effect upon the messages structure.
This loose coupling, intrinsic in the Document style, leads to more robust and reliable architectures that are easier to maintain, and having a modular aspect.
This is why it is better to design service at a fine-grained and composite them together.
If we do so, we can easily achieve the immutable service and any changes will only be done on the composite services, version..etc.
We will not have any management problems with composite services because we can discard any of them and rebuild them from scratch without even affect the underlying services.
The resistance from the IT is the among one of the major problems that cause and led to SOA failure.
SOA is a concept and best practice, vendors cannot do more than educating and training but the best practice and concept have to be followed by someone in the organizations, except if we have the vendor take over all the activities from the organizations.
This is could be done by SaaS / cloud computing but the problem that this model will not be successful without having a good SOA infrastructure in place.
IT has to regognize that we had a paradigm shift since 2002 toward assembly not development
Re: Business Transformation
I certainly agree that loose coupling will enhance the ability of the Enterprise to replace components in principle. The key is to understand which components are causing the most pain and need upgrading or replacing. Like someone who needs a hip replacement surgery, the parts that are causing the most pain are the parts that the Enterprise is most motivated to replace. At a technical level, of course document style is more declarative and loosely coupled, and we've experienced a shift from RPC for that reason towards a more internet style doc style. The concern expressed by JJ is more about monolithic enterprise applications and the difficulty to update them. For example, there are still many users on SAP R2 and there will continue to be for at least the next few decades. The business logic trapped in these applications is very hard to replicate. So I wanted to support what JJ said that vendors do need to work on upgradability and do believe that this is a major effort at most vendors. So I think the debate is a legitimate one, just at a different granularity than you imagine.
I think a vendor can do much more than either "take over" all the activities from the organization or stand on the sidelines and educate.
Let's start with the premise that Enterprise software is for the most part intended to be deterministic (by which I mean that for the most part Enterprises value the stability and predictability of its behavior) and that 90% of the difficult unpredictability comes from the human side of the equation. Let take one example such as a software dependency that crosses an organizational boundary. In this example, the provider organization can change or decommission the software based on their own interests without consulting the consumer organization. This would result in chaos and downtime for business critical systems and perhaps a ripple effect, but in some ways the provider may not care about impact to consumers. Vendors can provide tooling that help trace dependency and impact so this kind of situation is at least clear to all the participants. It's like having a traffic light at an intersection--the cars can choose to ignore it, but the tooling at least enables and supports enforcement of a trusted framework.
Another example of tool support is tracing fault across organizational boundaries. If a user experiences service degradation in an SLA, they put in a support ticket. Well, if that service crosses multiple organizational boundaries, without the ability to trace fault to root cause in the context of the user experience, you will get a situation where multiple organizations will blame others and the fault will not be resolved. In these cases SOA itself is often blamed even if the fault lies in an underlying system, for example a database server goes down.
So I think it extreme to say that SOA is entirely a human social issue and nothing to do with technology as it is fashionable to say at the moment. It is also extreme to say that ownership of SOA Governance tooling makes you successful in your implementation. I have long asserted that one illustrative metaphor is the idea that owning a hammer and chisel will never make you into a Michaelangelo. However, you will not see Michaelangelo trying to carve the David out of marble with his bare hands.
If you agree that Governance is necessary to achieve SOA Adoption benefits, you should likely agree that Governance tools (yes, even those supplied by "evil" vendors) are necessary but not by a long stretch sufficient.
Perhaps another way to put is is that service interfaces enable you to hide complexity of implementation in exchange for trust. Initially you can have point to point trust in the form of contracts between individuals. You may even graduate to a culture of trust. But in order to scale you need a framework of trust. If you look again at the traffic system, you dont trust each and every driver on the road, you trust in the rules of the road that are legislated to keep people safe, but also the system that monitors traffic accidents and fatalities to create new adaptations to the laws, the system that regulates traffic such as traffic lights and the system that enforces the laws including police and all the radar, cars, computers and equipment they use. This is how you scale trust into a bigger framework.
I appreciate what you're saying, I just wanted to provide my point of view on some of those points.
My 2 cents,
Re: Business Transformation
>> This is why it is better to design service at a fine-grained and composite them together.
>> If we do so, we can easily achieve the immutable service and any changes will only be done on the
>> composite services, version..etc.
This is the absolute anti-pattern of SOA. If you think a second about what you have just written, you realize that when you have 500 consumers of these "fine grained" services, you just set up yourself for 500 changes when the changes need to happen in the composite logic. Creating services at the Data Access Layer level is the worst way to approach a Service Oriented Architecture.
SOA is about creating and factoring assets which support upgradability without always/necessarily breaking existing consumers. XML technologies and contract definition mechanisms such as WSDL or ebXML BP are at the core of enabling this upgradability.
Re: Business Transformation
You will not have a consumers to this fine grained services directly. These fine-grained services is composed into a business processes.
If there is a change in the composite business process simply, ignore it and build a new composite service from scratch which will not affect the fine-grained services that were used in the first expired composite.
The fact that objects are serialized and deserialized to and from an XML stream, most structural changes can be introduced with zero-impact.
I did not mention any thing about the creating services at the data access layer, I am still in the high-level view, although there should be some services of course on the data access layers too when we dig down into technical details.
For example, in a document/liter wrapped style, we might have a an outcome class that has two attributes: returnMessage and returnCode.
There are some consumers who bind to it.
We will add a new attribute "other", to it.
The server module can now assign a value to the new attribute and the clients that update to the new outcome class can receive that value.
But what about the clients who do not update? Well, they will continue to work in the same way.
They will obviously not be receiving the new value, but the deserialization process will not break.
On the other side, we can remove an attribute (for example, returnMessage) in the server module, and have a non-updated client receive a null value in this field, though still working.
The document/literal wrapped style offers a new paradigm for thinking different than the one that we have got to use it, the RPC one.
Re: Business Transformation
Having just got off the phone with JJ (on a previously scheduled matter involving a large financial institution) I am inspired to weigh in on the conversational thread here.
One of the great frustrations of SOA Architects in this day and age is the discussion of appropriate architectural patterns. The reason for the frustration is that even after 10 or more years of effort, we continue to have conversations where the architects seem to pass each other like ships in the night. One of the main reasons for this is the efforts to scope granularity of a given pattern.
Composing fine-graned services is generally now referred to as "Orchestration" rather than "Business Process". The original confusion may have arisen because of the use of BPEL ("Business Process Execution Language") for such orchestrations. Unfortunately, BPEL or other low level integration orchestration languages rarely result in anything that a business person would recognize as a "Business Process". Interesting, the orchestration of low level technical services using capabilities such as BPEL often produce single coarse-grained services that are referred to as "Business Services". Once you reach the business level of granularity, composition of these higher level components can then be proposed as "Business Processes". But unfortunately, there's even another level of granularity to consider, which is the concept of whether the process is within or across a single department or business unit. As soon as a "Business Process" crosses a business unit boundary, it must be treated differently (from an optimization and continuous improvement perspective) than a departmental business process.
I say all of this not to further confuse readers, but just to say that one of the most accursed and frustrating things about architectural patterns (as opposed to say, object oriented patterns) is that differing granularities frequently call for different patterns and that the blanket statements that Architects so love sometimes turn to mush as you span multiple granularities.
I write about this problem in my blog post "On the Inside of an SOA, it's too Dark to Align with the Business"
So I respond to vent a little frustration, but also to hopefully advance the cause of Architectural consistency by awakening people to this communication problem. Lets compare apples to apples please.
Of course there is an alternative to SOA
"I challenge anyone to either
1. Propose a viable alternative that addresses the problem of “Enterprise Architecture Syndrome” as defined below."
There is, and has been for a long time, an alternative with many successful implementations. Recently this has gone under the name Staged Event Driven Architecture (SEDA). You can read about it at en.wikipedia.org/wiki/Staged_event-driven_archi.... InfoQ has a very interesting presentation from the BNP Paribas team that built their market risk system (see it at www.infoq.com/presentations/Risk-System-Harper-...) which they (correctly I believe) characterise as SEDA. This processes 100 million transactions a day.
In my work I have used SEDA for a market regulation system handling 10 million transactions a day, the architecture of a huge capital markets trading middle office, the architecture of a large north American retail bank, and the architecture of a large European bank. I have a colleague using SEDA to re-architect a large European postal service.
However, SEDA is for the management of transactions once they have been captured into a record keeping system. It is not appropriate for managing how those transactions get entered in the first place (almost always by people). Remember that systems are about people interacting with systems and systems interacting with systems. SEDA is for the latter not the former. Actually, I find that most of the confusion within SOA is caused by a failure to make this distinction.
For people interacting with systems, particularly if they want to use an agent such as a Web browser for doing this, we have an architectural style that works very well, namely REST. This provides a style (that is, a set of constraints) that work very well for such interactions. REST replaces the previous successful architectural style that was introduced in the late sixties by IMS and CICS (both from IBM, and still running the majority of transactions in the world) called conversational or pseudo-conversational respectively.
I am increasingly introducing REST into application architecture in my work as an Enterprise Architect. It appears to work extremely well if the constraints are indeed complied with. The uniform interface constraint enables caching (all GETs are cacheable) which enormously improves scalability. The 'Hypertext as the engine of state', or HATEOS, constraint beautifully takes advantage of the big difference between people interactions compared with system interactions - that is, the presence of the world's best semantic processor, a human. A computer needs to be told the meaning of every data element, humans can always work it out from context. This eliminates the need for WSDL in these kinds of interactions.
Re: Of course there is an alternative to SOA
it's always nice to hear from you. I always find it refreshing to learn about your pioneering work on scalable systems.
It's hard for me to comment on SEDA, as I would need to study it at greater length. The results from BNP Paribas look very compelling and I enjoyed their talk on the topic greatly. I can see how scalable the data architecture is and think it's a great achievement. I enjoyed learning about their heatmapping paradigm and their ability to visually display quantitative information such as load. It's a great result, certainly worth more study.
As far as REST, I think you said it yourself wonderfully well. REST in my view is an "application architecture" and not an "Enterprise Architecture". I agree that REST is astonishingly scalable, as we can see from the dominance of the URI encoding system that's been so successful on the Internet. You won't get any argument from me in that respect. But the challenge of Enterprise that I put forth in my article, above, is the challenge of traversing organizational as well as technological boundaries. REST can easily traverse technology silos, but from a perspective of Enterprise Architecture needs a bit more oomph to start to encompass some of the organizational issues faced by Enterprise.
I issued the challenge in earnest, so I'm very perfectly happy to be proven wrong and to eat crow. I enjoy a healthy debate and I issued the challenge on InfoQ because I value the community here and think there's a lot of cultural value placed on this kind of debate.
My 2 cents,