BT

Microservices and the Big Ball of Mud

by Mark Little on Aug 10, 2014 |

The concept of the Big Ball of Mud has been around for many years and we reported about it back in 2010. The concept is nicely summarised in this article too:

A BIG BALL OF MUD is haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle. We’ve all seen them. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.

With the rapid rise in discussion and use of the Microservice concept, it wasn't going to be long before architectural discussions around microservices, SOA etc. turned to the Ball of Mud. Recently Gene Hughson and separately Simon Brown both wrote articles on the subject. Simon starts by discussing how most of the industry is building traditional monolithic architectures, which can quickly turn into Balls of Mud:

On the one side we have traditional monolithic systems, where everything is bundled up inside a single deployable unit. This is probably where most of the industry is. Caveats apply, but monoliths can be built quickly and are easy to deploy, but they provide limited agility because even tiny changes require a full redeployment.

Then at the other end of the spectrum you have service-oriented architectures, which we've reported on for many years and which hopefully most of our readers are familiar with by now. As Simon states, this approach:

[...] buy[s] you a lot of flexibility and agility because each service can be developed, tested, deployed, scaled, upgraded and rewritten separately, especially if the services are decoupled via asynchronous messaging. The downside is increased complexity because your software system now has many more moving parts than a monolith

Now in Simon's view microservices represents a middle-ground:

We can build monolithic systems that are made up of in-process components, each of which has an explicit well-defined interface and set of responsibilities. This is old-school component-based design that talks about high cohesion and low coupling, but I usually sense some hesitation when I talk about it.

Simon refers to how Karma recently posted about how they have re-architected around Microservices and which we reported earlier too. In the original Karma article they mention that with their old monolithic approach everything got "entangled" and this was one of the reasons they decided to re-architect. However, Simon does not believe that is a sufficient reason to make the switch and perhaps even that microservices won't help with the problems that caused the entangling problem in the first place. As he says:

If you're building a monolithic system and it's turning into a big ball of mud, perhaps you should consider whether you're taking enough care of your software architecture. Do you really understand what the core structural abstractions are in your software? Are their interfaces and responsibilities clear too? If not, why do you think moving to a microservices architecture will help? Sure, the physical separation of services will force you to not take some shortcuts, but you can achieve the same separation between components in a monolith.

And Gene agrees, using a slightly different analogy to try to make the point:

Someone building a house using this theory might purchase the finest of building materials and fixtures. They might construct and finish each room with the greatest of care. If, however, the bathroom is built opening into the dining room and kitchen, some might question the design. Software, solution, and even enterprise IT architectures exist as systems of systems. The execution of a system’s components is extremely important, but you cannot ignore the context of the larger ecosystem in which those components will exist.

Simon has discussed the spectrum of architectural approaches with various groups and finds that those building monolithic systems don't want to consider component-based design. He discussed how earlier he had run a workshop with one of the teams and they attemped to produce a diagram of one of their software systems:

The diagram started as a strictly layered architecture (presentation, business services, data access) with all arrows pointing downwards and each layer only ever calling the layer directly beneath it. The code told a different story though and the eventual diagram didn't look so neat anymore. We discussed how adopting a package by component approach could fix some of these problems, but the response was, "meh, we like building software using layers".

He concludes that if teams have trouble building well structured (well architected) monolithic systems how are they going to be able to adapt to, and architect well, microservices? He agrees with Michael Feathers, who hosted a panel session at QCon NY earlier this year and who wrote about it, which included "There's a bit of overhead involved in implementing each microservice. If they ever become as easy to create as classes, people will have a freer hand to create trouble - hulking monoliths at a different scale." As Simon says, distributed Balls of Mud are things we should all be worried about.

It seems that Simon hit a chord because the comments on his article appear to agree with him. For instance Ralf Westphal writes:

[Microservices] are just another type of container. So as long as you have difficulties structuring your software using the "lower level" containers like components, libraries, classes, functions... you won´t be able to reap many benefits from µServices. Where there are monoliths today, where there is no experience with components, µServices will actually make things worse, I guess.

Pieter H also writes:

Yep, the second you introduce distributed, you need to leverage infrastructure that addresses network latency, fault tolerance, message serialization, unreliable networks, asynchronicity, versioning, varying loads within the application tiers etc. etc. Otherwise you're coding it yourself, a la NetFlix OSS and I suspect that is one of the main reasons for monolithic shops not being interested. Takes top level talent at the moment, not something all enterprises have access to.

Michael Groves raises an interesting question when comparing microservices and CORBA:

When we were building CORBA systems in the 90's, we talked about a lot of the same benefits as the Micro-services folks are now. Then the industry seemed to cool on distributed objects, even Martin Fowler declaring "Don't distribute your objects", mostly because of latency. Wondering why distributed Micro-services are good, but distributed objects are bad?

Finally, Kevin Seal finds microservices a more natural way to develop and believes it's more than a rebrand of existing approaches:

When a team chooses microservices I think it's important to acknowledge they have at least made a choice. This alone might be the reason their project turns out better than a more de facto monolithic design. For this reason I think it's important to have "trendy" topics so that they can get teams talking about what they're doing.

Perhaps only time will tell as to whether microservices offers a better approach and away from the Big Ball of Mud, or it represents a way of tying together (micro?) Balls of Mud.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

CORBA vs. microservices by Konrad Lukasik

CORBA has object brokers through which business logic is accessed - those brokers hide physical location of objects and might lead to performance issues, because developer might not differentiate between local and remote calls. Of course this is possible with microservices, but less likely.
Moreover CORBA maintains brokers, counting its references, this is statefull by design and for that reason less scalable. Microservices, especially based on REST, are better in this aspect.
All above it based on my limited knowledge of CORBA.

Another lost decade of SOA is about to start by Jean-Jacques Dubray

After trumpeting REST was the Solution to all distributed system architecture problems, our industry is yet again coming up with a "new" way to build distributed applications. Now we have Swagger and generate code from it, or worse client side SDKs.

If you don't know how launch a new distributed system architecture pattern, let me highlight the recipe:

1/Claim it's different and solve your biggest pain point
it is different than traditional SOA and, more importantly, it solves many of the problems that many organizations currently suffer from.

you feel the pain? we have the remedy, it's different ... yes REST too was different, and supposedly you didn't need any contract, client side, code, ... pure nirvana

2/ Create a picture of the pain point that everyone can relate to
The (sometimes evil) monolith

for REST, it was WS-Death-Star, term coined by the venerable Stefan Tilkov

Yes, they even dare to bring back the "ESB" in the discussion,
one way to think about microservice architecture is that it’s SOA without the commercialization and perceived baggage of WS* and ESB.

why not, right? somebody must remember that TLA.

3/ Make sure science is on your side
Fortunately, there are other architectural styles that do scale. The book, The Art of Scalability, describes a really useful, three dimension scalability model: the scale cube

In the case of REST it was Roy's thesis, but here they could not find something of that caliber, so if someone writes a book about the Art of Scalability, it must be good.... right? Incidentally, Roy's Thesis was gold for the RESTafarians as they knew nobody would read it, and let alone admit they didn't quite get it. So after HATEOS, we now get the Scale Cube, life is wonderful, isn't it?

So we all know where this will end up, people who don't use an ESB (aka a mediation layer) to take control of their service interface end up coding one. Over the last 15 years, I have seen that very real pattern repeat over and over, there are two main reasons for that:
- point-to-point (aka microservices) does not work. What they don't show in the figure 3 of that article is that is your customer service interface services changes, some consumers don't want to see the change, they are just fine with the "service" as it used to be. This will kill you, there is no other way around, repeat after me: "point-to-point" does not work. REST was point-to-point, worse, REST was distributed objects, and for the same reason REST does not work because there is no mediation layer in its architecture.
the cure to point-to-point is mediation, and yes, contracts, the contract-first kind

- microservices are an antipattern to consistency, the event-ual kind. Again, if go back to figure 3, you have a customer service and an order service, but, when you request some order data, in general, there are some customer data that need to come up, for instance a small piece of information such as the customer address (I know you can ship to a different one, but say you moved, wouldn't you want your default shipping address to be updated? or do you ask your customers to key-in their new address a second time?)
SOA and distributed system architecture is about eventual consistency, consistency from where you get your source of truth, consistency from which all the systems of record are eventually updated with the correct information

If you don't understand eventual consistency, you can't architect distributed information systems. Period. It's actually ironic because Roy himself got mad at distributed transactions, and called a few people a bunch of names, when all there is to distributed information system is distributed transactions. When you update the customer address, you indeed need a service that orchestrates the change of the customer address **AND** the default shipping address. This is not very hard to understand, yet, neither the RESTafarians nor today the microservices guys get (eventual) consistency.

So, when the same (kind) of people, come up with the same kind of arguments, will you listen? ... beware of the snake oil ...

Balls of Mud are a logical problem, not a physical by Hermann Schmidt

Question: As of today, has everyone managed to keep the dependencies of components inside a monolith under control, yet? For instance with a tool like Structure101? No? So why are we asked again to distribute this mess over the network now?

The Ball of Mud is the logical problem of coupling components. The coupling chaos gets potentially worse the smaller the components are! This must be obvious for any developer who is struggling with, say, class dependencies.

This doesn't make any sense to me. From my experience, monoliths suffer from a missing vertical slicing, not so much from being too coarse-grained in their guts. Once the business domains are separated (the monolith is vertically sliced), we need to find a sensible, manageable size for deployment units. This cannot mean chopping everything in 100 l.o.c pieces, each of which has its own lifecycle. That is unreasonable in terms of versioning and dependency management.

Managing a monolith is a piece of cake compared to managing a distributed network of services. I'd rather start with chopping a big monolith into, say, five little monoliths first before I let it explode into 200 pieces.

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

perhaps it is time that people move away from the coupling and cohesion mindset to think in terms of intent and consistency.

How can you achieve any kind of meaningful cohesion when everything is related to everything? The "ball of mud" is the expression of the fact that data is relational, that there is indeed customer data in your product "vertical". No matter how you will turn the question and slice the architecture.

The role of a service is precisely to offer an intentional interface on the consumer side, and an easy way to orchestrate components to realize that intent, consistently. What most people don't understand, especially the REST/microservices guys is that there are two faces to a service, the back (consistency) is just as important as the front (intent).

The "back" is not visible because we are all trained in classical OO programming, we focus on encapsulation rather than dependencies.

The problem with vertical slicing, especially today when you need your services to work with your Web App, Table App and SmartPhone App (and tomorrow your Wristband app), is that it does not promote an intentional interface, quite the contrary.

Re: Balls of Mud are a logical problem, not a physical by Hermann Schmidt

I think I now understand where you are heading. The relationships are what causes the entanglement and that is a logical fact ignorant of any architecture.

The verticals cannot be perfectly isolated, of course, and are a compromise trying to achieve a good cohesion. We try to cut at the "thinnest" connections. They miss the intent, that's right. I might try to arrange them hierachically. Then, a vertical may orchestrate other "lower" verticals. However, this may not evolve well over time. Thus, some coordination services sit in front of the verticals to expose use case specific intent. Any distribution makes this difficult in terms of transactional consistency. That's why monoliths (meaning: big deployment units) are so much easier.

But what would it be without verticals? What is the decomposition? I mean, we had (and still have) that consistency layer. It is the (ACID) database. In a pure horizontally layered monolith, anyone can manipulate any entity and join things together across vertical domains. We know that this is difficult to scale. What is your vision in a distributed world? Is it duplication with eventual consistency?

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

>> Is it duplication with eventual consistency?
Yes, exactly. This is what happens in any enterprise today. You have anywhere between a few hundred to a few thousand "apps" which need to expose their data across several channels. This is a reasonable view of what a "mature" microservices architecture will look like. You can only think in terms of intent and eventual consistency.

So yes, you will need:
a) mediation, how can you build a distributed system without mediation? Don't you need to hide changes to components which don't want to see them?
b) orchestration (both Stateless for Queries and Stateful for Commands). This is the only way to achieve consistency (eventual).

Some people have tried using events in lieu of orchestration but that does not scale very well (development-wise) because complex exceptions and exception handlers can't be well expressed with a series of events. I am not saying, never use event, I am just saying don't write orchestration code with events.

The pattern that absolutely no one is talking about is the "double-face" pattern.A service has two faces. Everyone focuses on the consumer side, but the back side is just as important. I describe some of the concepts in this presentation: Service-as-a-Software (the Other SaaS).

To succeed at building a service architecture, not only do you need to mediate change on the front-end, but you must make it easy to on board Systems of Record within the Service implementation.

There is no "Autonomy" in SOA, services are not Autonomous. It's a fact, you can't fight it, you must embrace it.

It's time our industry grows up and stop using corny software engineering patterns (such as encapsulation, coherence and coupling).

If I learned only one thing when building Service Oriented Architectures is that the best Software Engineering patterns simply do not apply, they all lead to disasters. SO is not OO, not DO, not even close.

So let's stop talking about microservices, they won't work, and I don't want to lose another ten years explaining people that's a dead end.

Services are "flat", a thin layer between your channels and your systems of record. Vertical slices can still be implemented with that kind of architecture but they are more "bent" than vertical if you know what I mean.

Microservices, Reactive, SOA ... oh my! by Russ Miles

My take has been that we need software that is as close to being as adaptable as our agile processes and businesses need to be. It's this adaptability, supported through granularity and loose coupling, that is one supporting leg that can lead to what many businesses need; i.e. the possibility of innovation.

Microservices in my experience are a set of principles that underpin the need for software systems that can keep up with the pace of change needed. Reactive are similar, in my opinion, and so in fact if we stop arguing about microservices vs. X, and focus on what we want, things become much simpler.

My two cents.

Re: Balls of Mud are a logical problem, not a physical by Hermann Schmidt

Very interesting thoughts, although I cannot quite figure a concrete architecture with today's possibilities, yet. Especially the interplay between an orchestration layer operating on commands and eventually consistent distributed data.

You are right though, managing eventual consistency is the key here. When I let service A have copies of data, which may not be consistent with each other at all time, we may get funny results from service A, and on top of that even inconsistent results from service B, which has copies of the same data.

That sounds dreadful to me for clients, who can never be sure that anything matches. Alas, this is the nature of distributed systems, so, as you say, embrace that or revert back to centralization.

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

Yes, that is and will remain the key problem in microservices. You get "agility" at the expense of consistency. That's the CAR theorem, you can't have consistency, agility and relationality at the same time, you have to pick two. You can be agile and relational, you can be consistent and relational, you can even be consistent and agile, as long as you don't have too many relationships to traverse, say like a social app. But, I don't believe there is a way around CAR. Most companies who have applied vertical slicing have reached a deadly wall, where for any MVF they pick the "minimum" and never get to the "viable". That's where microservices land, it looks easy at the beginning, and then one day everything becomes hard, that day you are dead. Again, a very real pattern, that I have seen several times at play.

There are two key problems that explain why our industry keeps moving circularly when it comes to distributed system architecture:
- people believe that the programming model is a given. We have been using OO for nearly 30 years (25 for me) and we don't see it as the problem. Think about this: the key aspects of information (identify, state, relationships) are all reified as class properties. Most of the code we write is boiler plate code that simply add these missing semantics to OO. And guess what, functional programming won't be of much help either.
- mediation and orchestration don't emerge easily in traditional programming models, again, they are orthogonal to OO, there is no mediation in Java or Scala, there is no orchestration either. Yet, 90% of the code written is actually some kind of mediation and/or orchestration, but totally unfactored.

If you want to tame change and introduce the right level of agility you have to make change possible by:
a) having a component in the architecture that hides change to whoever does not want to see it. It is not change that is the problem, it is the undesired propagation of change that is the problem. Making change "easier" is not going to solve that.
b) when you enact a new relationship in a system of record (no matter how small the SoR is), you need an architecture component which is going to help you reach the right level of consistency with the smallest amount of code possible, and that's what an orchestration engine does for you.

Everything else: REST, microsoft services, coupling, cohesion is complete bullshit (pardon my French). Don't you guys understand that the programming model IS-THE problem? not the protocol, not the granularity with which the assets are designed? We when from SOAP to raw HTTP, what did we gain? nothing, we are back to contracts and generated code. We went from RPC to message oriented and now back to RPC (talking to overlapping datasets). Only the insanes in our industry can believe that Microservices will deliver any value.

Re: Balls of Mud are a logical problem, not a physical by Jeppe Cramon

There is no "Autonomy" in SOA, services are not Autonomous. It's a fact, you can't fight it, you must embrace it.

It's time our industry grows up and stop using corny software engineering patterns (such as encapsulation, coherence and coupling).

This is quite interesting. In my experience autonomy, encapsulation and coherence can be achieved for most types of services.

I'm wondering the difference is due to different definitions of services. You mention
Services are "flat", a thin layer between your channels and your systems of record.

From this it sounds like services, to you, are the coordination/orchestration parts that make existing systems able to coordinate. Is that correctly understood?

To me a service is a technical authority for a specific business capability. It owns all the data and business rules that support this business capability. To me this is the embodiment of encapsulation and coherence. If a service has the necessary data to perform its logic it will have a lower coupling to other services and thereby it can be autonomous. To support business processes we of course still need coordination (orchestration or choreography). These are IMO also services (one name for them could be process services) and they own the given process/sub process business capability. These services are not particularly autonomous (but can still have encapsulation of process state/logic), but they're quite interesting as they're the embodiment of business processes, which honestly is there most action/change is happening and where companies can differentiate.

So if I understand you correctly, the process/coordination service in my description resemble your definition of service?

Re: Balls of Mud are a logical problem, not a physical by Jonathan Allen

Each vertical slice is known as a "bounded context" in Microsoft's architecture documentation. They highly recommend the approach and one I personally agree with.

* Note: Microsoft did not invent the term, but I can't remember who did.

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

Jeppe,

unfortunately I believe your view of services is incorrect, though wide spread, you need to slightly change your perspective to understand my point of view.

First you need to move from business capability to intent
Second you need to think that Systems of Records are not part of services, there are two faces to a service, the intentional interface and the back of the service where systems of records are plugged in (for queries and commands alike).

Let me take an example, you just built a social app, it manages well users and posts. It's been successful and now you want to expand in commerce and start selling products to your users.

Obviously the "social network" team does not want to be coupled to the "commerce team", they want to release functionality independently of each other, essentially operate independently as much as possible. Yet they share customer information.

This is a very real pattern everywhere, in every company of any size, you need to plug new systems of record all the time, because companies grow and as you pointed out you need new business capability.

The mistake that everyone I know makes is to think that the interface to the system of record is the Service interface, that precisely Autonomy and Cohesion are valid Service Oriented patterns. Unfortunately they are not. Services live between the channels and the systems of record, precisely because to realize a particular intent, such as changing a customer phone number or an address will in general require the coordination across several systems of record. If you push this responsibility to the Systems of Record they all need to become coordinators. The reality is that they may even have different rules altogether. Your social app could be available world wide, but your commerce team only operate in a few countries.

If you start coupling the SoR together either in a monolithic way or in a microservices way, you will indeed create a big ball of mud.

When you create an intentional interface wired to coordinators you achieve a number of virtuous goals:
- intents a highly reusable, it's safe to call the "changeOfAddress" operation from anywhere
- Systems of Record are highly decoupled and autonomous from each other, it keeps your commerce team and social team very happy
- most importantly, you can now plug new systems of records easily at the back of your intent/coordinators

We all understand that you need different systems of record, this is even more true today with the prevalence of SaaS.

The questions I have for you are:
a) since you must achieve consistency (eventually), how do you do it?
b) if you choose to do it with coordination, where does this coordination live? on the consumer side? on the system of record side? somewhere else?

It's an honest mistake to think in terms of autonomy and cohesion, but it is nevertheless a mistake. What you must achieve is autonomy of the systems of record. This is what SOA is about. SOA is about creating a normalized interface to the systems of record.

I have discovered this pattern in 2004, published a book on InfoQ in 2007 that you can download here, and I am now working on an open source project called Chorus.js to bring this pattern to Node.js.

It's really not that hard, everything else you do that is not intentional/coordinated is pretty much RPC or even worse, Distributed Object.

We'll talk about processes another day, IMHO, business processes don't exist, the real concept between the activities we perform are Business Entity Lifecycles. BELs, IMHO are the only kind of business logic that need to live in the service layer, other than basic validation. But that's pretty abstract, it would require a much longer discussion. To understand intent/coordination it's best to keep the business processes out of the discussion. Just think in terms of consistency and Systems of record.

Re: Balls of Mud are a logical problem, not a physical by Jeppe Cramon

Jeppe,

unfortunately I believe your view of services is incorrect, though wide spread, you need to slightly change your perspective to understand my point of view.

I respect that you disagree and I'm open to try and understand your approach. Perhaps it's differences of wording, misunderstanding or basically that we different architectural challenges that form our mindset. My SOA views are largely focused on how to build future services that can get us out of having to do lots of RPC/orchestration and basically avoid the need to complex containers like ESB's. So take what I say as being focused on building new greenfield services rather than how to best deal with legacy integrations and trying to SOA enable them (some of the approaches can still work here but the playing field is different because boundaries are much more blurred and there are typically more systems that own or store the same data).
My services view is formed by the work of people like Bill Poole, Udi Dahan, Greg Young, Eric Evans to name a few. This isn't meant as name dropping, but more to explain from where my views have been formed. I've recorded a (rather lengthy) talk on SOA and EDA here www.tigerteam.dk/2013/slides-and-video-from-our... and have a running blog series on SOA here www.tigerteam.dk/category/soa/

First you need to move from business capability to intent

Second you need to think that Systems of Records are not part of services, there are two faces to a service, the intentional interface and the back of the service where systems of records are plugged in (for queries and commands alike).

Let me take an example, you just built a social app, it manages well users and posts. It's been successful and now you want to expand in commerce and start selling products to your users.

Obviously the "social network" team does not want to be coupled to the "commerce team", they want to release functionality independently of each other, essentially operate independently as much as possible. Yet they share customer information.

This is a very real pattern everywhere, in every company of any size, you need to plug new systems of record all the time, because companies grow and as you pointed out you need new business capability.

The mistake that everyone I know makes is to think that the interface to the system of record is the Service interface, that precisely Autonomy and Cohesion are valid Service Oriented patterns. Unfortunately they are not. Services live between the channels and the systems of record, precisely because to realize a particular intent, such as changing a customer phone number or an address will in general require the coordination across several systems of record. If you push this responsibility to the Systems of Record they all need to become coordinators. The reality is that they may even have different rules altogether. Your social app could be available world wide, but your commerce team only operate in a few countries.

If you start coupling the SoR together either in a monolithic way or in a microservices way, you will indeed create a big ball of mud.

When you create an intentional interface wired to coordinators you achieve a number of virtuous goals:
- intents a highly reusable, it's safe to call the "changeOfAddress" operation from anywhere
- Systems of Record are highly decoupled and autonomous from each other, it keeps your commerce team and social team very happy
- most importantly, you can now plug new systems of records easily at the back of your intent/coordinators

We all understand that you need different systems of record, this is even more true today with the prevalence of SaaS.


It's still a little blurry for me exactly how responsibilities and data ownership is split in this example. A drawing of call paths/message channels, what data lives in which SoR and how this relates to the coordinates would be very helpful :)
From what I understood and interpreted, the SoR sounds like Entity/Data services (or perhaps an abstraction that encapsulates the many systems that claim ownership of a specific piece of data) and the coordinators sound like Activity/Task services (if we relate it to the classic layered SOA pattern).
The Task service is where you place your intentional interface and it acts as a coordinator ensuring to update any SoR, so any user of this interface doesn't need to deal with consistency of SoR?

The questions I have for you are:
a) since you must achieve consistency (eventually), how do you do it?
b) if you choose to do it with coordination, where does this coordination live? on the consumer side? on the system of record side? somewhere else?

As I mentioned services to me are owners of a business capability, which means they own data, logic and UI (preferably) related to this capability. A services is composed of Autonomous Business Components (to stay in Udi terminology). E.g. see this presentation www.infoq.com/presentations/SOA-Business-Autono... at 50:20 and again at 1:01:00

There can only be one owner of a business entity (e.g. Customer), which could e.g. be called the Customer Service.
To me a service is similar to a Bounded Context, which means that during analysis we might identify that different services might have different perspectives on and relationships to an entity (e.g. Sales, Shipping, Inventory are all interested in product, but they're typically interested in very different data related to a product). This means that I wouldn't create a single Product service (or SoR) that contained the combined information on Product. Instead I will allow each service to store and control exactly the Product information it owns, because that creates smaller models which are inline with the responsibilities of this service (responsibilities of applications where services are used is of course much more across a lot of services, but this I prefer to handle using Composite Applications/UIs).
There will of course be other services that are interested in other Services data for different usecases, e.g. the Sales service may be interested in knowing a customers status (e.g. is the customer a VIP customer which should get a credit of X %).
We can either have the Sales service request this information from the Customer services on a need to basis (e.g. using RPC), which I would like to avoid, since I (like you) don't want the Services/SoR's them selves to become coordinators, as that creates runtime coupling and weakens their autonomy, encapsulation and coupling. Instead I like to apply Event Driven Architecture (EDA) principles (Publish/Subscribe of business events) which allow the Customer service to publish an event, e.g. called CustomerStatusChanged(State = VIP), which the Sales service can subscribe to and store as data internally in the way it best sees fit. This means that Sales services eventually will be consistent with the Customer status. The ownership of the customers status is still firmly placed within the Customer service, which is the only place it can be changed. I've shown a more thorough example here www.tigerteam.dk/2014/microservices-its-not-onl...


It's an honest mistake to think in terms of autonomy and cohesion, but it is nevertheless a mistake. What you must achieve is autonomy of the systems of record. This is what SOA is about.

It sounds like you get autonomy and cohesion for SoR and I get it for (my) services. But we get it in different ways, with different splits of responsibility and composition. The reason why I prefer encapsulating coherent data within each service is to avoid synchronous updates to multiple services (I prefer to handle this using events since they give me business context, idempotence and automatic retries), since it creates many complex issues like show on slide 65 in www.slideshare.net/jeppec/soa-and-event-driven-... - you could solve some of these using BPEL engines - but I prefer something (to me) more simple by using events.

SOA is about creating a normalized interface to the systems of record.

By normalized do you mean one company/enterprise wide canonical model, irrespective of usecase, that all need to agree on?
In my SOA approach I focus on avoiding canonical models (since this is dealt with using Bounded Contexts) and having services that expose RPC and async Command style interface externally to other services. Instead I like to use EDA to coordinate services and business processes. I've shown this in my blog link above, it's covered in slide 83 to 98 in www.slideshare.net/jeppec/soa-and-event-driven-... and it's also covered in this talk by Udi Dahan www.infoq.com/presentations/SOA-Business-Autono... at 27:12

Most often RPC style services are only needed for read usecases and typically in the context of some form of UI, e.g. on an invoice I want to print the customers name. This I would like to solve using Composite applications/UI's where my Customer service renders the part of the UI where the customers name is printed (alternatively if the customer name is part of the Invoice service because it was part of the InvoiceCreated event - then the Invoice service can render this information in a composite way).
I haven't had time to write up my composite UI yet (it's next in line), but it's both covered in the video taped talk and on slide 75 to 81 in www.slideshare.net/jeppec/soa-and-event-driven-...
Udi also has a very good talk on it here vimeo.com/44235655


I have discovered this pattern in 2004, published a book on InfoQ in 2007 that you can download here, and I am now working on an open source project called Chorus.js to bring this pattern to Node.js.

It's really not that hard, everything else you do that is not intentional/coordinated is pretty much RPC or even worse, Distributed Object.

I skimmed it through and read the parts that explains your decomposition, but to really understand your principle I would love to see a more detailed example. E.g. as a drawing that shown the responsibility decomposition and responsibilities and how things are communicated (messages/?) :)

We both agree that RPC and distributed objects should be avoided like the plague. So our intentions are the same, how we approach them is perhaps different (I'm not quite sure yet).

It's hard to discuss these details decoupled in writing. I'm talking at skillsmatter.com/conferences/6312-mucon and I would love it if you would suggest a talk on your approach. This would also give us a chance to discuss our (different) approaches together with some other great people :)

/Jeppe

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

Jeppe,

unfortunately I will not be able to come to London at that time.

Let's take the example that I was talking about, simple, but quite representative of what usually happens.

Unfortunately InfoQ does not allow the img tag so here is the link.

The scenario is that you have built a successful social network and you just hired a commerce team to introduce a new capability "buy what your friend recommend". You have already built a System of Record that contains Customer / Posts information, where do you put the commerce system of record? I think we all agree that it would not be a great idea to mix it up with the social SoR. You want the commerce team and the social team to operate independently of each other. The commerce team selects a payment service and a fulfillment service as well which are both accessible via simple APIs. The fulfillment service accepts a "Ship" request and bill your company monthly for all the orders. Similarly, the payment service accepts a "Charge" request and sends you every month a big chunk of money (we'll keep the reconciliation / reporting out of this example).

But the problem is that they all share "customer information" (name, address, ...). For Fraud reason both the Payment and Shipping service keep a mapping between customer and address to detect possible frauds.

Just to make things a bit more spicy, the social network operates world-wide but the new commerce capability is only available in the states, with plans to expand later.

So far so good, but right there you see how microservices break down, there are these pesky relationships that cross service/capability boundaries. How do you handle them? For instance, where do you implement a "changeOfAddress" operation? on both services? one? neither? What is the intent of the customer? change the billing address? the home address (for which he/she gets coupons). A good user experience would actually keep track about "links" between addresses, such that if you change one address, all the "linked" address get changed. To keep it simple, let's assume that the commerce team decides that the "default shipping" address will be updated automatically with the home address and an option will be provided to select "update my billing address" as well. Again, how do you implement this simple changeOfAddress intent? You orchestrate the update to the social SoR and to the commerce SoR.

Now you have been very successful, and you decide to introduce a new marketing product "Get a coupon for what you friends buy", you decide that this marketing team will operate independently of the social and commerce team, ... their engine is based on location, so knowing your exact zip code is key to them, how do you "plug" this new capability (and SoR) into your information system? Wouldn't you agree that if you had an "orchestrated" change of address if would be a piece of cake and neither the social nor commerce team would be impacted?

In SOA, CQRS is very important (I believe we all agree on that one), Queries pull information from the "System of Truth". If you do consistency right, most of them are, but for scalability/security... reasons you might pick one vs another, but you can never normalize your system of record, you can only normalize access to it. What you want is to achieve the autonomy of the system of record, and that is not achieved by creating "autonomous" services, quite the contrary.

The other patterns at play is mediation. On the consumer side of the service you have a number of consumers, each one with slightly varying requirements. For instance the "getCustomerDetail" operation is going to be different for social and the commerce team. Ideally, you want one operation that goes to the "system of truth". How do you handle that situation? First you have to understand that in SOA, reuse happens the other way around, it is not an operation that you build for one consumer that is reusable for another. When a new consumer shows up, it usually requires some small changes, so you change the implementation (as in get more customer details), reuse happens when the old consumer is capable of consuming the new version of the operation. Again, how do you architect that:
- you can use a compatible versioning strategy
- you can use mediation, i.e. you transform the new interface to look like the old, such that existing consumers are not impacted

I know both are a bit harder than what some people call an "immutable" versioning strategy (copy the code and start over for a new consumer), but I have shown here that this "immutable" versioning strategy will kill you.

Now, there are multiple ways to implement mediation and orchestration and we can debate what's the best way (and the reality is probably it depends), but we should agree that without mediation and orchestration your SOA, micro, nano, restful, whatever, will run into trouble, sooner rather than later. The reason why RPC and DO don't work is precisely because they can't do mediation and orchestration well, nor can JAX-RS/WS or any framework that wires interfaces to code. It just doesn't work.

This is why I recommend to focus on Intent and Consistency. If you focus on "autonomy" and "cohesion" you will get killed, one autonomous, immutable, cohesive operation at a time. These principles don't apply to Service Orientation because data is relational and relationality makes data not very "autonomous" and "cohesive".

What do you think?

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

To respond more specifically to your questions:

what I say as being focused on building new greenfield services rather than how to best deal with legacy integrations and trying to SOA enable them

There is absolutely no difference between the two, the problem starts when you have more than one System of Record. I hope that you agree that the example I took illustrate that the pattern works for "new services", not just the "old, legacy/integration".

avoid the need to complex containers like ESB's

Please don't put RPC and Orchestration on the same footing, you are doing a disservice to yourself. Since when an architect "avoids" something a priori and decide that because it is "too complex" it must be bad? Are we talking about a Skill Oriented Architecture?

The reality is that Apache has a bunch of these containers that don't look very complex compared to the complexity of writing mediation and orchestration code by hand, or fighting an immutable versioning strategy after a few months. I know that it does not sound as cool as Scala or Akka or even functional programming, but ... it's really effective.

the SoR sounds like Entity/Data services

An SoR is wherever data is persisted (database + data access layer). The SoR has its own integrity rules, data access patterns and semantics. It can be inherited, 3rd party, home grown, ...
There can only be one owner of a business entity (e.g. Customer), which could e.g. be called the Customer Service.

This is where the disconnect is, because practically, this can never be achieved. Never, the Commerce team will need to create its view of the customer that will be different from the Social team or the Coupon team. The infamous Customer object will grown in each system of record, over time, they System of Truth will become blurry and consistency will become more difficult, unless, unless, the service layer always route the queries to the SoT for any given piece of information and ensure (eventual) consistency across all SoR.

Instead I will allow each service to store and control exactly the Product information it owns, because that creates smaller models which are inline with the responsibilities of this service

Yes, absolutely, but the problem is the "relationships" right around the edges of the relationships will start denormalizing that perfect view. You simply cannot partition data, how could you? Relationality will always be in the way.
Instead I like to apply Event Driven Architecture (EDA) principles (Publish/Subscribe of business events)

Again, EDA principles are flawed in an SOA. An event is "the occurrence of a state". Sometimes, changes happen in an SoR outside the service interface, that's when an event is emitted. But you can never have services "subscribe" to events. Imagine an exception happens in one of the subscribers? how can you ever roll back? you publish another event? The response to events need to be coordinated (a special orchestration that only listens to events and coordinates the invocation of services, including exception handling and roll backs)

to avoid synchronous updates to multiple services (I prefer to handle this using events since they give me business context, idempotence and automatic retries), since it creates many complex issues

Many complex issues that become far less complex when you use the appropriate tool for it: orchestration. You are trying to solve problems with the wrong tools. Idempotence and automatic are good, but that's still does not help you with exception handling. By the time, you decide you are in an exceptional condition, you are stateful, you can't maintain context that long (reliably) in memory. Orchestration is the solution you are looking for (please look at chorus.js if you think orchestration is complex).
but I prefer something (to me) more simple by using events.

Well, as I said, events have a role to propagate the "occurrence of a state", but they cannot replace orchestration, period. If you think you do, you'll be playing ping/pong with your events for the end of time.

I'll look at your materials later this week, but it appears to me that you come out thinking mediation and orchestration are complicated and because they are complicated they should instantly be dismissed. I agree that there are multiple ways to skin a cat, but our industry has been at this question for a very long time, and we have tried a number of things that look all the same me: we tried to change the wire protocols and we tried to change semantics of the calls (Resource Orientation), and we got nowhere. Microservices will have the same fate, because they suffer from the same problems.

The only thing we have not tried is intent/consistency. Again, I am not even trying to tell you how to do that (I can tell you how I do it) but could we agree that service interfaces need to be intentional and achieve consistency? Then we could take a series of specific examples to show how intent/consistency can be achieved.

Re: Balls of Mud are a logical problem, not a physical by Vjeran Marcinko

JJ,

I've been following your posts around, I like vocal people even if some think of it as not being overly polite :),but as Jeppe mentioned, I too often miss some *very* detailed example with lot of pictures of whole architecture, lots of concrete messages and their channels drawn, to completely understand you. I guess probably some words are overloaded in this area, so people think of different things when these are written.

For example, one starts with your example, and you come to problem of changing the address, and your question where to put that operation ends with no thing drawn where it ends - in other words, is this orchestration separated as another service with "changeAddress" operation, and it updates all other interested services "at their backs" via some "SoR-related service interfaces"? Or its used just as lib inside some existing service which is considered SoT for customer? Either way, it would be good to draw it to easily understand that etc...

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

Vjeran,

I appreciate your comment and I am happy to spend the time it takes to make my position clear.

Here is for instance the orchestration I had in mind

The point that I am trying to make is that "relationships" make it impossible to create "autonomous" services.

As I said, I am certain someone will say, "But I can do that with events" and my response to that is how do you deal with exception and rollback? more events? Let's say paypal's fraud detection system returns an exception and say that it cannot change the billing address to the requested one. You've decided to trust paypal's fraud detection system and you now decide to roll back the address and notify the customer (at it's old email/home address).

The SoR-related Service Interface, should be called "integration points" such that we don't mix up what is a service and what is not. Generally SoR cannot keep their interface constant if any change is made to the data model. That's a key characteristic of Integration Points, not services. You do not design and implement services like you do for integration points. I would say this is the main error people do. Services are not attached to any system of record. IPs are always attached to a SoR.

Again, I am happy to give any detail you'd like.

Re: Balls of Mud are a logical problem, not a physical by Vjeran Marcinko

OK thanx, but one more question - I always considered Source-of-Truth the place where initial change is made, and all other interested parties are notified from from and accordingly updated. Of course, if we just take that simplistic event-based update approach here, here comes the problem that you just mentioned where some interested services cannot update their SoR because of some reason, so now we have to orchestrate this exceptional situation and one can quickly lose their head trying to make sense of all those events flying around with no firm context.

But in your picture, there is no place which is SoT for customer (or at least its address), right? Since update notification just came from this orchestration service, right? Its not that I have any objection about that, just to make it clear.

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

yes, that's a good practice but it depends on your requirements for eventual consistency. The problem is when scalability gets in the way (e.g. comments on InfoQ). Obviously InfoQ is using a different SoR for writing and reading comments. So where is the "source of truth"? Is it that important that everyone see a comment that is still in the "editing window"?

I would say at first the SoT was the Social SoR (Zip code), and over time, the Commerce System took over. You can even have different SoT for different pieces of data for the customer. It's not necessarily recommended, but that's the point of a service interface is that you decouple the interface (and hence the consumer) from the details of the implementation. All the consumer cares is that the service interface remains stable until it needs something else. In general a SoR cannot keep that interface stable, let alone hand off the Source of Truth to another SoR altogether.

As I said, you can never normalize "The Information System", you can only create a normalized service interface. That's the whole point of SOA, and yes, events are part of the picture, but they can't, alone, deliver that normalized interface.

Re: Balls of Mud are a logical problem, not a physical by Hermann Schmidt

Ah, that picture helped. I try to put it in my words:

The commerce and social is what I would have called a vertical slice. In your ecample, I understand that they are totally independent systems (monoliths even), and that is quite common in real life.

Then, to avoid scattering of the changeAddress operation across commerce and social, which would couple those systems in an unpleasant way (distributed coordination - yuck), an orchestrator aka service is introduced. It takes care of distributing the address data. It is responsible for the eventual consistency.

The points where the systems are attached to the orchestrator (the service) you call integration points, not services. Services present client intent, which covers more than one integration point.

The service may have to be adjusted to modified integration points. This may change it's "inner" interface.
The mediation layer's task is to shield modifications of the "inner" interface of the service from the "outer", client-specific interface.

Is that about right? :-)

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

yes absolutely, now you can control the size of each "monolith" to whatever makes sense from a business perspective, such that the commerce team and the social team are as independent as they can possibly be. It will never be total decoupling though.

If you want to be good at "change" (some people call that agile), you need to:
a) make it easy to implement new intents (that's generally covered by any approach)
b) be able to hide change to consumers who are not ready to see it (that's achieved via mediation)
c) and yes, you need to be able to plug easily new SoRs at the back of your services, if the "ChangeOfAddress" operation had been bolted on the Social SoR, how could the commerce team plug itself in the "ChangeOfAddress" service?

REST didn't work, we went right back to APIs, contracts, ... We got a pretty CSS (Swagger is much sexier to look at than WSDL, but it is exactly the same thing). The Uniform interface didn't survive (we have APIs), and of course Hypermedia was pure science fiction.

Because we are back to square one, a few people in our industry want to serve up a new iteration of "bolted integration points as services", that again will not work, no matter how you slice your architecture, up, down, lateral. It will not work because interfaces need to be intentional (not bolted to a piece of code) and the implementation need to be consistent, which again you can't do when you are inside a SoR, it's not the right technology/container, and worse you couple that SoR to another SoR because now the Social SoR need to know something about the Commerce SoR.

It used to be that an ESB (for mediation) and a BPEL engine (for orchestration/consistency not BPM) would cost you a couple of arms and legs, but that's no longer true, by a large margin. They are just as cheap as Node.js, all available open source on Apache.

If you are not familiar with Service Orchestration (again not BPM), please take a look at this project I am working on: Chorus.js.

Re: Balls of Mud are a logical problem, not a physical by Vjeran Marcinko

Hi again,

Just to clarify - mediator such as ESB is primarily used in consumer-to-service channel, not to mediate orchestrator-to-IPinterface channel? Or its used for both?

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

great question, I would actually recommend to mediate Integration Points as well. Systems of Records is that they are generally unable to keep an interface constant.

The main goal of mediation is to control how change propagates, whether it is from Integration Points to Service Implementations and from Service Interfaces to Consumers.

You could also decide to implement that mediation on the service implementation's side. It's only a question of level of effort: you have to weigh how much effort will it be to control change at the integration point level or within the service implementation. You don't want to spend as much time crafting the IPs interface, as you would do for the Service interface.

@Jeppe by Jean-Jacques Dubray

Jeppe,

I took a look at your presentation "SOA & EDA" on slideshare. I am not surprised you dropped from the discussion. I believe that you'll need a proper SOA reeducation to help you understand what consistency is and how you achieve it. There is not a slide that would actually make sense in the entire presentation. I would recommend that you drop the stock photography and focus understanding what it is you are talking about. I'll publish a response to your presentation shortly.

It's sad that to see that SOA is still so misunderstood in soon to be 2015, and because of that just about anybody can put a couple of funny pictures in a presentation and self-proclaim himself or herself a SOA expert.

JJ-

Re: @Jeppe by Jeppe Cramon

Wow, just wow. What shall I say to that? That arrogance runs in your veins?
I will not fall to your level; I was under the impression that we were in a constructive discussion about different styles of SOA - where you of course you have the right one ;-)
I'm sorry if you feel let down but my lack of response, but I had more important tasks that needed to be completed in the mean time.

Re: @Jeppe by Trond Hjorteland

Sorry to say JJ, but I think you just hit yourself with that punch. I really enjoyed the argument/conversation, especially since it highlighted the big differences between the classical SOA (what I often refer to as the Erl school) and the more logical version (mostly seen in .Net land and by some referred to as SOA 2.0 or event-driven SOA).

Re: Balls of Mud are a logical problem, not a physical by Jeppe Cramon

I will give it a quick attempt at responding to some of your comments/questions, though I fear it's pointless after your message from today. Given the generally different and at times confusing views on and definition of SOA it's great that you have the right answers. So let's try.

Please don't put RPC and Orchestration on the same footing, you are doing a disservice to yourself. Since when an architect "avoids" something a priori and decide that because it is "too complex" it must be bad? Are we talking about a Skill Oriented Architecture?

The reason why I mention them together is that most Orchestrations I've come across have been the coordination of RPC calls (SOAP/REST/etc.) to other systems/services.
The orchestration you linked to in the response to Vjeran looks very RPC like (even though it may be based on a messaging approach such as Request/Reply?)
For certain integration that's the best fit. My argument is not if orchestration works, but instead that that it's one of the patterns that can be applied. I know you disagree and that's okay :)

the SoR sounds like Entity/Data services

An SoR is wherever data is persisted (database + data access layer). The SoR has its own integrity rules, data access patterns and semantics. It can be inherited, 3rd party, home grown, ...


Which sounds equivalent to the Entity/Data service in layered SOA (I'm trying to find a common language). It's not clear to me if SoR can contain business logic (like calculations, etc.) or if it's pure data + data constraints/rules?
Your SoR and service design sounds like something along the lines of bill-poole.blogspot.dk/2008/02/service-boundary... and bill-poole.blogspot.dk/2008/02/why-single-servi... (with good business alignment), but it could also sound like an application of the layered SOA which is how most SOA implementations (with ESB) that I've come across is following bill-poole.blogspot.dk/2008/05/layered-service-...

Can you give a precise definition of split of responsibility (or difference) between SoR and Services? Who owns what, who allowed to do what?

There can only be one owner of a business entity (e.g. Customer), which could e.g. be called the Customer Service.

This is where the disconnect is, because practically, this can never be achieved.

That was a misstatement on my behalf. I meant to say there should only be one owner of a specific piece of information for a given entity (the SoT for this information). Customer information can definitely be split across many services (or SoR's in your vocabulary) as long a they transactionally don't need to be consistent and therefore updated at the same time :-)

Never, the Commerce team will need to create its view of the customer that will be different from the Social team or the Coupon team.

I think we agree on this. To recap what I've previously written: "To me a service is similar to a Bounded Context, which means that during analysis we might identify that different services might have different perspectives on and relationships to an entity (e.g. Sales, Shipping, Inventory are all interested in product, but they're typically interested in very different data related to a product)."
I know DDD can be very flexible on what a Bounded Context is - so e.g. Udi uses the name Business Capability to define the Service to business alignment. The general problem with Services is that everything easily becomes a service, so I find your distinction on SoR interesting (and perhaps it's closer to Autonomous Components).

The infamous Customer object will grown in each system of record, over time, they System of Truth will become blurry and consistency will become more difficult, unless, unless, the service layer always route the queries to the SoT for any given piece of information and ensure (eventual) consistency across all SoR.

I think we agree on this. For many usecases I use the UI composite pattern (which is also a form of orchestration/coordination) to pull data from different Services to create a (for the given use case) coherent view on data from different services.
To keep Services in sync, in case more than one service stores the same piece of information, or they need to react to business events from other services, I prefer to use Event choreography combined with typically a Process Manager (so there's someone noticing errors/exceptions) but I also use/combine it with Orchestration when it fits. So eventual consistency is definitely in play.

Again, EDA principles are flawed in an SOA. An event is "the occurrence of a state". Sometimes, changes happen in an SoR outside the service interface, that's when an event is emitted. But you can never have services "subscribe" to events. Imagine an exception happens in one of the subscribers? how can you ever roll back? you publish another event? The response to events need to be coordinated (a special orchestration that only listens to events and coordinates the invocation of services, including exception handling and roll backs)

I will say that you argue against a lot of qualified SOA experts with great conviction, I give you that. There are EDA patterns eventually combined with different assignment of responsibilities , that I believe solves some of the concerns you rightfully bring up. EDA has been applied to SOA many times with good result (I believe Udi even calls them two sides of the same coin), since they form/influence each other.

I like people that challenges my views because that's how I evolve them.
Your attitude seems to have changed from constructive to rude. So unless you are open to constructively discuss where we in reality align and/or differ, then I don't think it's worth going into further details.

/Jeppe

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

To be frank, I am not sure who is the most arrogant, is people who continuously claim SOA is broken, come up with their Fad du jour, be it REST, microservice, EDA or whatever, just to realize a few years later that it does not solve any problem.

The core of the problem in your reasoning is that you are mixing two things: the protocol and the semantics you want to carry over a protocol, you call things RPC, Event, Request/Reply. Let me tell you a big piece of news, SOA is not just transport independent, it is protocol independent.

Why is it that RPC does not work? have you spent even a second asking that question? because in the end the consumer has to make a call of some sort to something on the other side. RPC does not work because it is brittle, you change the method signature, existing consumer can't call anymore. You could also have some scalability reason or reliability reasons, but fundamentally, any approach that wires a consumer call directly to the business logic on the service side is bound to fail, it is not a question of language, a question of protocol, it is a question of decoupling the consumer from the service, hence you need an interface that has properties that are unknown to most software engineers. For them an interface, is just... well, an interface. In a distributed system you need an interface that can be mediated to avoid breaking existing consumers when it is not necessary. You can achieve that in two ways:
- mediation (but it multiplies the number of interfaces that points to the same service implementation)
- compatible versioning, where the consumer and the service provider agree on some compatibility rules that will allow the interface to evolve without breaking existing consumer

That is the cure to RPC. Interfaces MUST BE intentional. The Service implementation MUST BE decoupled from its interface and yes and ESB such as Apache Synapse is a great, simple and free way to do that. You can make fun arrogantly, as much as you want, at ESB, but you are simply shooting yourself in the foot and everyone else who listens to you.

The second mistake that you are making is that you absolutely want to classify "services". Again, I have news for you, there things called "Integration Points" that are exposed by systems of record and services that call these integration points, yes, in the end, it has to end up with an RPC call, there are only so many layers that you slide in until actually have to hit some business logic. In SOA you only need one decoupling, between the consumer and the service implementation. Below the service implementation, well it depends.

The third mistake that you are making is that you are enable to understand the semantics of Service Oriented programming model, you don't understand what an event is, how it used. An Event is the Occurrence of a State. If you use that official definition it will always work, you will never get lost. The beauty of that definition is that it finally brings the notion of State in the programming model. You are constantly mixing up the concept of a "message event" and an event. Sure when an event occurs, you need to let other components know, so a message needs to go somewhere, but that's a message not the event itself. Then you mix the protocol, you say the best way to propagate a message event is publish subscribe, therefore EDA is the answer. I have news for you, this is complete boloney, you need to decouple the semantics of the programming model, from the protocol, from the transport of the protocol.

So no, Orchestration is not RPC, by a million miles, the arrogance is to come with this kind of unsubstantiated bullshit and even think that this could be true. The arrogance is to come with EDA and think that a protocol is going to solve anything. That is incredibly arrogant when you don't even know the formal definition of what an event is, how events relates to the lifecycle of entities, how orchestration relates to that lifecycle and distributed transactions (not the 2PC ones, which we all know don't work), how distributed transactions relate to consistency and how consistency relates to intent and how intentional interfaces decouple consumers from service implementations and ultimately the system of record. When consumers and SoRs are decoupled, only then can you achieve true agility.

Telling people that "reuse" does not work when you don't even understand how reuse works in SOA is again pretty arrogant. I have some more news for you, in SOA, reuse happens the other way around, it is not the new consumers who reuse existing services, it is the new versions of services that are reused by old consumers. When a new consumer comes wishes to consume an existing service, it is rarely consumable as it, it will request some changes, these changes will break existing consumers. SOA is about enabling change in a highly distributed environment, change can only happen if you are able to hide it from components which do not wish to see it. Then and only then reuse will appear, but not the way you see it.

So again, I would recommend that you spend some time studying SOA, you simply know nothing about it, not a thing, and I am frankly tired that for 15 years there has been a number of self-proclaimed experts who all come with the same story: 1/ SOA does not work 2/Look at this big mess 3/I have the solution, it is XXX. SOA does work, we have everything to make it work, if only people like you would let us do our work.

Re: Balls of Mud are a logical problem, not a physical by Jeppe Cramon

To be frank, I am not sure who is the most arrogant, is people who continuously claim SOA is broken, come up with their Fad du jour, be it REST, microservice, EDA or whatever, just to realize a few years later that it does not solve any problem.

I admire how you think there's an agreement on what SOA is. If you had the courtesy to listen to the soundtrack you would have realised that the reason why I say SOA is broken is because that's what I hear companies complain about. All the companies I've worked with and for have had big SOA pains: they've had developers that didn't want to participate, they have experienced instability due to bad contract practices (like versioning), poor performance due to RPC (most orgs see SOA as Webservices and an ESB and then we're done), problems with centralisation of responsibility because the ESB was in the middle and this created many organisational issues, etc.
The next part is an exemplification of how SOA is practised in most companies I've come across and the problems it causes and that the solutions they usually thrown after it (add more power to the ESB cluster) doesn't solve the root problem. If you had listened to the sound track you would also have heard that I don't claim that the EDA approach shown is the best, but that it's a totally different way of solving the same problem (for read scenarios - write is a different matter which I get back to later in the talk). So that's the context.

The next I will say is that I suspect that you didn't read much/any of my other material, perhaps see/hear the presentation to the end. I've covered the issues with RPC in my online material several times. From my CQRS talk it should hopefully be clear that my understanding of events are (gosh) in line with what you write.

The rest of your "Jeppe didn't understand list" is a pain to read. I kind of feel sorry for you.
Sadly you didn't take the time to test your assumptions of my approach/understanding. You just assume that your interpretation is correct and go on a ramp saying "that's wrong"/"you don't understand"/"I've got news for you", etc.
Remember when pointing fingers at others, at least 3 fingers point back at you.

If you read more of my material, yes I know that takes time, you would realise that we agree on much of what you write and I do get the distinctions. And if the material didn't cover distinctions you find important, have the courtesy to ask if the lack of this represents a certain view, if it's an omission and it will be covered later. Also don't do a disservice to yourself and take one partially read/understood presentations angle/focus. There's a limit to how much I can cover in 1,5 hour. This is also why I said that discussing this in person would be a good option.
I would have loved to discuss why/if/where our approaches agree in some/more places and why I tend to favour a different way dealing with service reuse/contracts (my approach focuses on minimising the number of interfaces and their nature exposed to others - and thereby that amount of versioning required) on equal terms where we both assume the other isn't an idiot/amateur/.... or whatever your description of me indicates.

But this thread has lost its constructive tone. Instead of discussing big balls of mud, it has just ended up in mud throwing.

I'm out.

/Jeppe

Re: Balls of Mud are a logical problem, not a physical by Jean-Jacques Dubray

Jeppe,

I don't find the pattern: 1/SOA is broken 2/look at this big mess 3/Listen to me, I have the solution particularly constructive. You are, like many in our industry, using it, perhaps with less talent than other. Some have use "Same Old Atrocity", "WS-Death-Star", ... but in the end it is the same pattern. This pattern is the problem, this pattern is what makes discussion impossible, because you have the answer.

I did read your presentation. Reading your material is really painful, it makes no sense at all, I don't even know where to start. Your level of understanding of the space is close to zero. And no you don't agree with me, you simply don't see the articulation between all these concepts. Don't you say somewhere that reuse doesn't work?

All you care is that the conclusion is: EDA, publish / subscribe. Do you even understand what is an Entity Lifecycle? and how events as the occurrence of a state relate to them?

I have seen many painful SOAs over the last 15 years, and the problems always boils down to not understanding (in that order):
- the decoupling between interface and implementation
- distinguishing between integration points and services
- versioning
- consistency via orchestration
- what is the role of events in a SOA and the use the coordinator pattern to respond to events

People want to hear that SOA/distributed computing is as easy as a method call. Unfortunately it is not. It is not very complicated but it is definitely more that a = remoteCall(b).

So I repeat, SOA is not broken, we have all that is needed to be successful at SOA, all it takes is to understand what you are doing, with enough precision, which you obviously dont care about.

inspired by microservices by Alexander SAMARIN

- #BPM for software architects – from monolith applications to explicit and executable #coordination of #microservices architecture improving-bpm-systems.blogspot.ch/2014/08/bpm-f...

- #BPM for the #digital age – Shifting architecture focus from the thing to how the things change together improving-bpm-systems.blogspot.ch/2014/08/bpm-f...

Thanks,
AS

Follow up by Jeppe Cramon

FYI: The discussion continued online docs.google.com/document/d/1Vt96dy3HUzSPD42k6lJ... + twitter.com/clemensv/status/507188489163923456. So for those interested I've created a follow up slide set that attempts to both respond to what I believe are misinterpretations (perhaps due to lack of a sound track) and also sum of the different opinions on SOA that sparked from the online discussion that followed. I've added links to discussions, articles, videos and blog posts that I hope can be of benefit to all interested in SOA & EDA - www.slideshare.net/jeppec/soa-eda-follow-up

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

32 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT