00:25:33 video length
Bio Ganesh Prasad has 25 years of IT experience as developer and application designer; as architect in Shared Services he pioneered simpler and less expensive corporate IT strategies. He has written about Presentation ("Life Above the Service Tier"), Integration ("Practical SOA for the Solution Architect") and most recently Identity and Access Management ("Identity Management on a Shoestring").
Software is changing the world; QCon aims to empower software development by facilitating the spread of knowledge and innovation in the enterprise software development community; to achieve this, QCon is organized as a practitioner-driven conference designed for people influencing innovation in their teams: team leads, architects, project managers, engineering directors.
Eric’s full question: Eric Newcomer at QCon 2012 in New York with Ganesh Prasad, an architect who’s been working in the banking and telecom industries; he’s here at the conference to speak about identity management, but he’s done a lot of interesting work in the past around SOA as well; Ganesh welcome, could you spend a few minutes just to introduce yourself to the audience?
Sure; thanks Eric; I’ve been in the IT industry for about 25 years now; I started as a programmer, a COBOL programmer on mainframes; I went through three generations of technology, minicomputer era, Unix and then the client server era with Windows PowerBuilder and Oracle backends, and then of course web and Java; so I’ve seen four generations of technology but interestingly the same patterns of problems again and again and the same patterns of solutions as well; I’ve been a programmer, a solution designer, a project leader, and lately an architect for the last ten years and also in the last ten years, I’ve worked in the area of shared services at a bank, an insurance company and a telco.
Yes, okay; actually I came to SOA in an indirect way because I’ve been looking at shared services as I said and shared services are enterprise utilities; so these are technical services that are not business domain specific; so you find the same sort of things in banks, insurance companies, telcos, airlines, mining companies and so on; so they are things like identity management, integration, SOA, content management, communication gateways, document composition, things that are not owned by any single business unit but which are used by all of them and these present some unique challenges especially around funding.
Sure; they are called enterprise utilities in some places and sometimes they are libraries, sometimes they are standalone systems, but they are shared by different business units.
Eric’s full question: Where I was working previously at Credit Suisse, we had a long debate about sharing and how to share code; we had done some analysis to find there was a lot of overlapping and duplicate functionality in the applications and we could rationalize those down to perhaps a single function that would be shared, but then we had a big question of how should we share it the best way; should we share it with libraries, should we put an abstract interface on it, how should we do it; what are your thoughts there?
For me, the biggest challenge has been around not in deciding what needs to be done but how it’s funded and quite often everybody agrees that we need to move in a certain direction but a huge debate breaks out over who’s going to pay for that piece of work.
And that actually brings me to SOA and loose coupling because necessity is the mother of invention so as an architect in charge of shared services and having to implement a roadmap for implementing those shared services, I've had to think about creative ways of breaking up the problem into smaller components and positioning each one, marketing it so to speak to each business unit in such a way that a business unit sees some value out of that little chunk and then they are willing to pay for that; because the problem with shared services is either you need a big enterprise budget or the first project that comes along has to pay for the whole lot and neither of them is really feasible.
Eric: That’s very interesting because where I was working at Credit Suisse we had this idea that if we broke the problem up of let’s say modernization of the legacy environment into small enough pieces, into modules and components, it would be easier to move things forward toward new technologies.
Eric: Similar thinking or it was just, really just the funding?
Eric: So that was a part of it as well?
Yes; so it is more about trying to break down a larger piece of work into smaller components that could be deployed at different rates; so you couldn’t always say which one would get deployed first because one business unit would come up with a budget far faster than another; so even though you felt that this piece deserved to be done first, you’ll find another piece being done first; so you really have to architect the system in such a way that no matter in what order these components were developed, they could still plug together and eventually form this larger enterprise utility.
Eric: And this was successfully done?
Yes in more than one place; I’ve had mixed success because it’s not so much a technical problem but an organizational one.
Eric: Cultural perhaps even, how things are budgeted, organized, how people think about projects?
6. Since we have touched on SOA, I guess I should take a step back and ask what your definition of SOA is since there’s so many out there and so for you what does it mean; I know you’ve done a very well-read white paper called Practical SOA, can you explain your definition of SOA and what it means to be practical?
Okay; I found that in the industry most people conflate SOA with technology. And to my mind, it shouldn’t have been called SOA because that sort of led people down the wrong path; it should probably have been called DOT which I call dependency oriented thinking. And, after all, it's just a way of – it’s an organizing principle rather than a piece of technology.
You can apply this logic at every level of the enterprise; So if you're looking at different systems and they needn’t be technology systems, you need to look at what are the dependencies between these systems, what are the legitimate dependencies between them; so if you find that there are a number of dependencies that aren’t legitimate then you should look at removing them and the ones that are legitimate you should look at formalizing them; so you know exactly what a change to one system will mean to the other system; that is dependency oriented thinking.
Eric’s full question: I really think about SOA pretty much similarly to what you said, which is that it’s a design pattern or it’s an approach to a design or a blueprint for how you would redesign your IT environment and maybe develop applications a little better, more agilely with breaking the problem up, reusing functionality more easily, modularizing not really – and you can implement it with a variety of technology; so perhaps it’s important to separate the design from the implementation of this.
Correct; and in fact, I would say this applies above technology as well at the business level; so a very useful model that I have to look at enterprises is what they call the BAIT model; they have four layers, business, applications, information and technology and technology really is the lowest part of that stack and quite often the technology solutions fall out of your design once you’ve gone logically down those four layers; so even at the business level, you can apply dependency oriented thinking; you can say if we have a dependency on a partner organization, what exactly is it and can we formalize that in the form of some contracts; so we can eliminate the unnecessary dependencies that exist between our organizations and the part that is legitimate we formalize into a contract so that we are protected; we know exactly what will happen when there is a change in the environment and we can protect against it.
Integration is another interesting term; so let me be a bit provocative on this; so when you point your browser at a website and the page loads, what do you call it integration; I mean it seems so trivial that no one would ever call that integration; whereas when we have a six-month project that costs a few million bucks and we have a huge team working to get two systems to communicate and at the end of it you manage to get some interoperability between two systems that weren’t designed to work together then you call that an integration project.
To my mind, it’s a little outrageous really because I believe that that is a failure of prior design; systems must be built to be interoperable; so it’s got to be a Lego block kind of approach to building systems and integration should just fall out of the way systems are built, not something that you have to do over six months spending so many millions of dollars and say that’s integration and pointing a browser at a website and having a page load is not integration.
Eric: As long as they’re built in a consistent way according to standard practice or with standard technologies or standard design approach; because as you know if you’re going to an environment with a lot of legacy systems that weren’t developed that way, then you have a challenge of how do you make these things interoperate or integrate moving forward.
Eric: But if you are starting a new project then everything should be done with that in mind.
Well I’d say it applies to the way you approach legacy as well.
Because going back to dependencies right, so if you look at two functional areas, this was way back in 1974 I believe that there was a paper written by three people whose name I forget but this paper for the first time introduced this concept of cohesion and coupling; now if you look at it, service-oriented architecture is nothing but the application of this principle of cohesion and coupling; so what is cohesion; it’s just a principle that things that belong together should go together; so what forms an application is just a set of related functions and coupling is a principle that says systems should exchange the bare minimum information with each other; so you’ve got to reduce the amount of communication between two systems, two cohesive systems to the absolute minimum; if you apply this principle across the board, you will have SOA and you will have interoperability; it just falls out of that approach.
Eric’s full question: Why do you think there’s so much skepticism still about SOA; I know at the previous company I was working, Credit Suisse, there had been a very large SOA project at one time, which seemed to have failed, at the least the perception was that it failed, and people as you had said earlier confused technology with the design pattern and said SOA is too slow which is obviously not correct for a design pattern, but they seemed very reluctant to try it again because of failure; so have you run into this skepticism?
Oh, absolutely; and there are two reasons for this; one is that when it succeeds, it’s not recognized as SOA because it doesn’t have any technology in it, it was just good design; and when it fails, it is clearly marked as SOA because it was technology; so I have seen examples of people using very expensive ESBs to connect two systems and they were under the impression they were doing SOA because they were intermediating two systems; and at the same time they were exposing internal data from one system to the other so they were actually tightly coupling these two systems at the data level and yet they would have been shocked at the suggestion that they were not loosely coupled because hey, we are using an ESB; so if that project failed because of that tight coupling, they’ll go SOA has failed but actually it failed because they failed to apply SOA.
Eric: In this case, I dug into it and it turned out the failure was due to deploying a service on a very lightweight Windows machine in an unsecured area and there was a dependency on it, a very mission critical service; when the Windows machine went down, the mission critical service failed and therefore they blamed SOA but this was really a case of inappropriate design or inappropriate execution of the design.
Yes, perhaps but one could also argue that one of the dependencies that existed wasn’t properly catered for. And so it was a SOA failure.
10. I had talked recently with a large manufacturing company and they told me they viewed everything as a service when they started on their SOA program; […] have you seen this or do you think this is part of the issue?
Eric’s full question: Well that’s fair enough; I had talked recently with a large manufacturing company and they told me they viewed everything as a service when they started on their SOA program and they took 3270 screens from their mainframe interactions and craft them as a service and they had a problem because things weren’t responsive, they weren’t working as correctly as they felt they should; there maybe it was just a lack of understanding of associating a function with a service instead of trying to just wrap every existing application and treat it as a service; have you seen this or do you think this is part of the issue?
Quite often. I think if we go down from the business layer towards technology, I think a lot of these problems will be solved; because at the business layer you need to ask yourself what are we doing, what are our primary business functions and then what are the processes that these business functions entail and then what are the steps in these business processes; and when you do that, you automatically come up with a set of cohesively defined functions that should be your services; now that is your target state of what a service oriented organization should be and we should be looking at how do we get our existing applications to move to that state. So I don’t know if that level of thinking had been done layer by layer.
Eric: I don’t think so; I think they just said let’s do SOA; everything we have is a service therefore we’ve done SOA as opposed to let’s do a design which is as you said what really needs to be done.
You know, this is really sad because it’s really hard to point to an example of success in SOA not because they are rare but because they are just not recognized as SOA when they succeed; SOA is just a question of sensible design where someone says let’s not do this because that’s going to couple two systems too tightly together; let’s only exchange the bare minimum information; that is an example of SOA and that actually saves money, it saves time down the track but no one ever recognizes it.
12. I agree with you that SOA is a very good and helpful design approach because it’s functional and it’s easy to relate to the business […] how can we be sure we’ve succeeded, how can we be sure the investment has been …?
Eric’s full question: I agree with you that SOA is a very good and helpful design approach because it’s functional and it’s easy to relate to the business, the functions that a business performs for its customers and other businesses; but we in our SOA projects, we’re always asked how can you measure the value of the investment that’s a little bit more investment in doing SOA because you have to create a shared service which is a little bit more complicated than having the same function exist in a single application; how can we be sure we’ve succeeded, how can we be sure the investment has been …?
Of late, I’ve seen organizations that when they cost a project they also cost the estimated savings they’ve got because they’re using reusable services so that’s a good start, that’s a good approach; but I think we should also take into account the less glamorous aspects of design where you just reduce dependencies; I mean we don't have to have created a SOAP service to say we’re reusing a service, just reducing dependencies between systems by not exchanging a piece of unnecessary information or not exposing a piece of internal data, those are examples of SOA thinking too and these are going to save you money in the long run; but we never ever account for them when we estimate future projects and we go "You know what? We made a really good design decision back there that’s saving us thousands of dollars today", we never do that.
Eric: Right; so the benefits of good design should be obvious and should carry through over years.
Eric: But it may be difficult to recognize the way projects are done and the way people think about IT?
I have a real provocative take on this as well. Because I have a beef with the industry definition of SOA governance. So there’s this view of SOA governance as a subset of IT governance which in turn is a subset of corporate governance. And corporate governance is supposed to apply to the governance of corporate assets, IT governance to the governance of IT assets and therefore SOA governance to the governance of SOA assets because SOA is seen as a subset of technology.
Eric: Where there are some additional artifacts SOA introduces into the IT environment, the interfaces, messages.
Well yes but my take on this is SOA is an adjective that says you are aware of dependencies; so when you talk about SOA governance, you’re basically saying I do governance with a consciousness about dependencies; to me that brings us to what’s the difference between governance and management.
Eric: Right; so this is a scope issue perhaps?
Well yes, yes and no; a lot of people use the word governance when they mean management; so this is one of the first things we need to tease apart, what do you mean by governance and what do you mean by management; so to my mind governance is about doing the right thing and management is about doing things right; so one is the what the other is the how; so one talks about objectives and scope as you said, the other one is how do we get there from here; so they’re two really different things and it’s like the board of an organization versus the executive management or like the project steering committee versus the working group; so one of them deals with objectives, what should be done and the other one deals with how do we do it.
So if we say SOA governance and SOA management, we’re really talking about dependencies; SOA governance is what are the right dependencies that should exist and SOA management says how do we get to a state where these are the only dependencies; so how do we eliminate unnecessary dependencies, how do we formalize the legitimate dependencies, how do we prevent fresh dependencies from creeping in so that’s SOA management; so SOA governance is all about saying what are the legitimate dependencies at each level, business, application, information, technology.
There are four levels; you need to even have committees at the business level to say what are our dependencies at the business level and unfortunately we just don’t have that kind of thinking at a business level.
It’s really a pity that SOA is something that potentially saves so much money; it improves your agility because when you reduce dependencies between systems, you can change systems without having to worry about impacts to others so you can move faster; your operational costs are lower because your related changes are less and you have to do less regression testing and your operational risks are lower because there’s less likelihood of you breaking other things when you change something; so what’s not to like; the business should actually want to adopt SOA across the board but unfortunately SOA has been seen as this kind of technology-only thing and a subset of technology at that and that’s why it’s failed to reach its potential.
16. All of the vendors have tried to coopt it; I remember very clearly when BEA came out and said SOA equals JAVA 2EE. Another vendor said have done similar; what about REST; we’ve heard a lot at this conference about REST web-based approaches; do you think that fits within the context of SOA or something else?
Absolutely, absolutely; and again unfortunately I blame the REST guys a little bit for this because they have often conflated the word SOA with SOAP-based web services; so they say REST is really good we can do all of this whereas with SOA you can’t; what they actually mean is with SOAP-based web services you can’t do this.
Eric: Okay, confusing technology with the design patterns.
Absolutely. And could – to the extent that REST also decouples say resources from representations and so on, it’s a form of SOA and again you can do non-SOA REST; I have seen examples of RESTful URIs that go something /customers/1234 where 1234 is the primary key of the customer in the database; so you are exposing something that’s internal to your domain out through your interface and you are tightly coupled now; what’s the point of saying, oh, I’m doing REST and so it’s loosely coupled; it’s not.
Eric’s full question: I agree with you; we should be able to consider REST as part of the toolkit for implementing SOA; I think we would have viewed it that way too and Anne Thomas Manes as you may know has a good talk on that about RESTful SOA where she makes a very similar case; in her view the main benefit is abstraction however it’s implemented whether with web services or REST; you abstract things and as you said if you’re putting in your primary key of your data that you’re looking up in your URI then you’re not abstracting either.
Yes; so I would say rather than have RESTful SOA, we need SOAful REST; we need to apply these dependency oriented thinking to any technology that we use.
Eric: I think everyone is doing something with the cloud and there are many definitions of that as well, but I tend to think of it as a virtualized environment the Amazon, Rackspace that kind of thing and do you see a correlation?
Oh, absolutely; let’s put on our dependency oriented hats again and look at the cloud, right; what is it that it’s really doing in terms of dependencies; I think that it’s not going to make, cloud is not going to make so much of a difference to developers, solution designers or architects; it’s actually going to make a difference to project managers and the business for the simple reason that it’s completely eliminated or driven down the cost and the time to provision infrastructure; if you look at any significant organization and any significant project, you’d find that the project managers are devoting a lot of time to provisioning infrastructure.
It starts at the beginning of the project and it goes on for about six months, how do we procure hardware, how do we deploy hardware, what do we do about the production environment, the DR environment, the test environment; so they’re actually looking at procuring servers, you know, installing them and getting them set up; that takes months and costs so much money; whereas with the cloud it’s a matter of minutes you can spin up new servers; so that is a major dependency that has gone away because of the cloud; so it’s an infrastructure provisioning dependency that the cloud has taken away and that’s the SOA view of the cloud in my opinion.
19. One of the things that I’ve heard in the financial services industry here in New York is everyone would like to move to commodity data center type of architecture with the cloud virtualization on top of it, […] is there something SOA can help with in that transition? Do you think that’s an issue?
Eric’s full question: One of the things that I’ve heard in the financial services industry here in New York is everyone would like to move to commodity data center type of architecture with the cloud virtualization on top of it, for those reasons easier to provision and spin up and spin down when you don’t need; but I heard of many people say that the big challenge is how do we get our applications well better suited for that environment; how do we get our applications moved over; is there something people need to do; is there something SOA can help with in that transition? Do you think that’s an issue?
Well, modern applications typically what is a server, it’s an IP address and most of them run on Java web servers or most of the modern software that we’re talking about runs as easily in a cloud environment as it is on on-premise environments; so I really don’t see any difference there in terms of what the cloud brings to us; I really see that removing a major dependency in the infrastructure provisioning side, which is where I think it’s going to have major benefit.
20. If I understand correctly, your main point about SOA and dependencies is that good design is going to have long-term benefits and probably would apply to the cloud, to internal infrastructure, anything so this should be done regardless.
That’s right and it’s not just technology, it’s anything. Any two systems.
This is actually a very good case study for how one can apply loose coupling to a shared infrastructure, to shared services right; so identity management is an enterprise utility; more than one business unit can take advantage of it but nobody wants to pay for the whole thing; so how do you break up identity management into loosely coupled pieces that can be chunked and funded independently by different business units and still be put together so that it forms a full-fledged identity management system; so that in fact was the subject of my white paper called Identity Management on a Shoestring that’s published by InfoQ and it’s available for free download and I also spoke about it at this conference.
So it’s basically around data design; so again I made the point that identity management is not about technology; the technology is the easy part; if you don’t have the right technology you can roll your own, but if your data model is wrong no matter how much technology you throw at the problem, you won’t solve it; so it’s more about some principles around loose coupling that you apply to data, how you keep, how you model the right entities, don’t use surrogates, use meaning-free identifiers, don’t expose internal identifiers to the outside world, use master data management principles to keep replicas in sync and so on; so it’s the really data oriented, common sense principles of cohesion and coupling really or SOA and that’s what gives you the loose coupling and the ability to independently fund chunks of this identity management system that you can all put together after four or five years and go you’ve got an enterprise utility.
Eric: I’m picking up a common theme here which is the importance of design to almost any kind of work that’s being done in the IT environment today.
22. Certainly, we heard about this from the speaker from Apple about the importance of design for Apple products which is very famous, but perhaps for these kinds of enterprise applications the importance of having a good design isn’t as well understood as it should be.
That right; I think we put too much faith in technology and we don’t think enough about the application design; I think that should come first; architecture should precede product procurement.
The recipe for Shared services is actually easy
My answer here.
Re: The recipe for Shared services is actually easy
You misunderstand the context in which I used the term "chunks". It is more coarse-grained than you have understood it. To take an Identity Management example, one "chunk" could be a system that does Single Sign-On for customers, with provisioning being done manually. The second "chunk" could be a system that leverages some of the infrastructure built in the first phase (e.g., SSO servers, directory, database, etc.) and extends it to provide support for internal users. A third "chunk" might leverage what has been built over the two previous phases and provide automated Windows-based logins to internal users. Each such "chunk" addresses the business need of a particular business unit, and so they see the value in funding that part alone. This way, we can build an entire enterprise utility in a loosely coupled way, with incremental funding, and after a few years, we will end up with an enterprise system that works together, yet is not monolithic.
A dependency-oriented approach will help to break up a multi-year implementation into the right "chunks" that fit together well. And this approach works in a real-world organisational situation, so I'm not talking theory here.
I don't understand why dependency-oriented thinking is somehow at odds with thinking about life-cycles. If the fault-lines lie around life-cycles, then those form your finest-grained functional units. In other words, a life-cycle is your measure of cohesion (or "sticking together" - I didn't use the term "coherent", which means something else altogether). You then build your systems around these cohesive units, with loose coupling between them (lowest possible levels of dependency).
What is the contradiction? I'm not even talking technology here.
Re: The recipe for Shared services is actually easy
Perhaps a better answer to your post would be:
Service reuse is not a fundamental objective of SOA.
Elimination of functional duplication is.
There is a subtle difference.
If you're saying that an operation cannot be reused out of its life-cycle context, then it can't. If there are two versions of it in different contexts, that's not a duplication. It's legitimately two functions. If someone is trying to force reuse of a service operation out of context, then they're being unaware of certain life-cycle dependencies. So there's no contradiction between what I'm saying and the point you're making about "atomic" life-cycles.
I always get the sense that we disagree and argue unnecessarily, and this is probably because we use such different terminology. To be frank, I sometimes think you're speaking French ;-).
Re: The recipe for Shared services is actually easy
The problem I see with that analysis is that the actions, the "calls" cannot be reused independently of each other. This is a blind mind's view of the code elephant. You make a couple of assumptions, that are simply unfounded:
a) calls are atomic in nature (whether they have side effects or not), what I mean by that is that the only thing you consider is the call and nothing else
b) calls are the unit of dependency
This paper about coherence and cohesion is single handled responsible of all the troubles our industry has endured for the last 40 years.
The real view of the problem is that
a) the lifecycle is unit of dependency, and not just the interface, because you can call in just about any order you want. Even when you make a single call into a lifecycle , you are dependent to the entire lifecycle (remember everything has a lifecycle).
b) calls are not at all atomic they are interdependent via the lifecycle, you cannot think about a call in isolation of the others
The real view of a system is an assembly of entities which each have their lifecycle. As the system receives external events (input from users, external systems, ...) some transitions are exercised in the systems and actions are invoked across entities, until nothing else needs to be done.
As long as you will be ignoring the very nature of what you are trying to reuse, you will never be able to reuse it, or more likely the reuse will be hit or miss. It is not about packaging what you are trying to reuse under some vague abstract principles, it is about directly understanding what you are reusing, that call is not just any call it is a call to a specific lifecycle and without understanding how this call relates to both the lifecycle and the context in which you want to reuse it, you have no hope whatsoever to reuse it. There is no amount of cohesion, chunking, dependency that will get you there, period.
Do you understand that when calls are related (interdependent) via a lifecycle there is no way to organize them differently? the lifecycle is the very justification for the calls. You have no room whatsoever to change them.
Re: The recipe for Shared services is actually easy
the only element of wisdom I am trying to add to the discussion is that an operation / action / call, however you want to call it, can NEVER EVER be thought about in abstraction to which it is tied to. This is not a "design pattern", this is physical, an operation / action /call is the visible part of the lifecycle. What you are reusing always is the lifecycle. An operation/action/call materializes a transition from one state to another, always. If a call X needs to be made to transition from State S1 to State S2, and now, in a different context you need to transition directly from S1 to S3, then you cannot "reuse" X. Without understanding the lifecycle and understanding how it aligns with your context of utilisation you have no chance to "reuse" anything.
Well, to be frank, if you think in terms of lifecycles to, why not say it clearly. When you say "If there are two versions of it in different contexts, that's not a duplication. It's legitimately two functions." I am not sure we agree on what function means. You seem to always speak in a language about what is reusable is very atomic, almost at the call level, and I claim that the only thing that can be reused is the lifecycle as a whole. It imposes legitimate dependencies between operations/actions/calls and there is no amount of cohesion that will remove these dependencies.
I have taken analyzed countless examples where people ended up creating "services" that were not reusable, and always been able to trace the problem to be missing states and/or transitions for a new context of utilization.
To the best of my knowledge, lifecycle, state and transition are not "French". They are clear, well established concepts, yet, they appear to be 'dirty' words...
Re: The recipe for Shared services is actually easy
1. You agree SOA is not about technology, and in the same breath, you talk about "calls". Perhaps you have not understood the level of abstraction at which I am talking about dependencies. Dependency analysis is just common sense, and it occurs irrespective of "services" and "calls". If the point you make is valid, rest assured it would be uncovered by the dependency analysis.
2. The 1974 paper by Stevens, Myers and Constantine is about "cohesion and coupling", not about "coherence and cohesion". Please look up and study the concept of "cohesion and coupling" before attacking it. You are probably using it every day without realising it.
Re: The recipe for Shared services is actually easy
No disagreement here.
> if you think in terms of lifecycles too, why not say it clearly
Because I'm talking about an approach that can and should be applied at different levels of abstraction, i.e., business, applications, information and technology. The view of life-cycles belongs in the business and application layers, and also influences the information layer. It's a legitimate part of dependency analysis, but not the only consideration. If you think purely in terms of life-cycles, how would you capture business layer, i.e., commercial dependencies (e.g., We move up to a higher charge rate if our service usage crosses such-and-such a volume)? How would you capture technical layer dependencies (e.g., Adding a field to a data structure can be very complex in IBM's IMS)?
Bottomline: Your point about life-cycles is valid, but it is a subset of a four-layer dependency analysis.
> I am not sure we agree on what function means.
By "function", I mean a business function, like "Loan Origination" in banking. This is a standard architectural term, so I'm not making it up. A function can be supported (implemented, if you will) by one or more processes. Some of the sub-processes, e.g., a credit check, can be reused, and some cannot. As you have pointed out, we need to analyse the possibility of reuse *in context*. If we don't, we are omitting a dependency, and it will come back to bite us.
> I have taken analyzed countless examples where people ended up creating "services" that were not reusable, and always been able to trace the problem to be missing states and/or transitions for a new context of utilization.
If I were to analyse those problems, I might have found dependencies that had not been taken into account :-). Same issue, similar analysis, different levels of scope and perhaps different terminology.
Re: The recipe for Shared services is actually easy
maybe we are passing each other, but it seems to be that there is a need to redefine cohesion and coupling, and hence reuse.
Coupling is currently defined by “the measure of the strength of association established by connection of one module to another"
Cohesion currently "refers to the degree to which the elements of a module belong together."
The paradigm shift is to understand that reuse means "assemble". In an assembly, the coupling is zero (the same module can participate in as many assemblies as possible) and the cohesion is always 100% (the thing that is assembled is perfectly cohesive). Cohesion and Coupling do not describe and do not apply what happens at the assembly level, hence at the point of reuse. I don't dispute the fact that these principles would apply below the assembly perhaps at a more technical level.
>> I might have found dependencies that had not been taken into account :-)
I am quite bothered by the "past tense" of your statement, which again is typical of the cohesion/coupling/reuse mentality of the 70s, that still prevails. You seem to indicate that there is a level of analysis that I would qualify to be as "ultimate", in other words, there is a level that you can reach that is able to support any potential evolution of the context, 3 years from now, 10 years from now, 100 years from now, whatever the context has become, I will be able to "reuse" the service I designed in 2012.
Don't you think again that you need to change your perspective and that in SOA, just like in real life, in the functional world that you describe, reuse happens the other way around? It is not the new which reuses the old, isn't it the old that reuses the new? In other words, reuse simply means having the ability to change something without breaking anything that depends on it, not building something from old components ... they are ... hum ... old. The problem is not a dependency analysis (that's paralysis), the problem is evolution.
The paradigm shift is:
reuse = capacity to evolve elements of a given assembly without breaking other unrelated assemblies using that element
coupling = capacity of an element to evolve while maintaining the compatibility with all the assemblies in which it is use (0 means any change you make has zero consequences, 100 means nothing can be changed without changing other components)
cohesion = degree to which an element can participate in an assembly without any other components, what I mean by that is given an element X, whatever element Y, there is at least one assembly where Y is not required to be. If every assembly where X is, Y also needs to be, then X is not cohesive enough.
If we can agree on these definitions, we would have made a giant step "forward".
Re: The recipe for Shared services is actually easy
In our discussions, I always get the sense that we are talking about very similar things, but we never seem to agree :-(.
"Reuse" should not be seen as a prime objective of SOA, so I don't understand why you keep harping on that aspect. Functional reuse is often a *by-product* of sensible design. Long before we come to technical components or modules, we should have done a dependency analysis of business-level functions and processes. Any functional reuse we obtain will come out of that.
I don't agree that cohesion and coupling are a 1970s relic just because the paper was written in 1974. These are enduring principles that we use even today for modularisation.
> You seem to indicate that there is a level of analysis that I would qualify to be as "ultimate" [...] there is a level that you can reach that is able to support any potential evolution of the context, 3 years from now, 10 years from now, 100 years from now
You have clearly misunderstood me. If we analyse a system and understand the dependencies that exist between its subsystems today (and these are at all four levels of the BAIT stack), then when the context evolves, we will immediately see what dependencies have been impacted by that change. Obviously, some changes will have an impact and some won't, but without a clear understanding of dependencies, we can't tell.
Am I making any sense at all?
Re: The recipe for Shared services is actually easy
I was really centering the discussion on Eric's question around reuse, and "SOA" is a failure. I don't disagree that SOA has other important aspets. People often, if not always point out that Service X coule not be reused and then procede to build their own. I don't know if you have heard of the Gervais Principle, but I view this particular point -reuse- as one of the core problem as to why organization cannot be reformed. There are others, but without reuse most companies lose velocity in their ability to adapt to a changing environment. They keep starting from scratch in everything they do, and then they die when their IT budget is entirely consumed to keep the light on on all the past solutions they built, that they can never turn off.
A lot could be said on the relationship between IT and the ability of an organization to reform itself, I happen to believe that the center piece is "reuse", which has 3 components:
a) capacity to assemble elements in a functional way
b) capacity to use existing elements, in new assemblies, with other elements (new or old) without causing change to these existing assemblies
c) ability to change existing elements as needed without causing change to the assemblies in which they participate
I personally don't see how the principles of Coupling and Cohesion apply in that context. Software engineering seem to always want that its oldest principles prevail. Don't you think that this is precisely the problem. Take versioning for instance. How many companies have implemented an immutable versioning strategy as opposed to a forward compatible one? Immutable may have made sense in the 70s in a world of libraries, but it does make sense in a Service Oriented world.
>> Obviously, some changes will have an impact and some won't, but without a clear understanding of dependencies, we can't tell.
But from a lifecycle perspective this is not a "dependency". When retail and private banking have slightly different lifecycles, this is not a dependency on the context, you can interpret it at your own risk, but we are only looking at the extensions of existing lifecycles, say of a particular investment. A lifecycle is self standing, it describes the behavior of an element, which can be assembled in any way you want. There is no dependency between a lifecycle and its context, at least this is what you need to strive for.
Loose coupling with REST
Thanks for the valuable insights.
on the subject of people exposing database primary keys through rest URIs (as your e.g.: customers/1234), what are your recommendations on practically overcoming these sort of dependencies.
I fully understand the idea behind the suggestion of not exposing internal information as a public interface, but I cannot think of any practical approaches to solve this. As you suggested, one approach would be to have some sort of a global mapper that do not depend on the primary keys map these Ids to domain specific primary keys. in this case those global ids could be used in the URIs instead of the db primary keys.
But I'm not sure whether this is practical for every single resource exposed as a REST URI as we would essentially be having a mapping layer on top of all the resources exposed.
Another approach that I could think of is using an identifier that has a semantic meaning with respect to the resource being exposed. (for e.g. the customer full name for the example above). but the problem is that it's not always possible to guarantee the uniqueness of the URI schemes, whereas with a generated ids we can guarantee the uniqueness.
What are your thoughts/suggestions on this?
Re: Loose coupling with REST
This is actually not hard if the database design has been done right. Let's say we have customer reference numbers that are expected to be given to customers to refer to themselves. This will form a _candidate key_ of the Customer table in the database. The actual database primary key for the Customer table will usually be an automatically generated sequence number. When exposing the URI to clients, it's the customer reference number that should form part of the URL and not the database primary key.