Ian Robinson discusses REST, WS-* and Implementing an SOA

Recorded at:

1. Hi. My name is Ryan Slobojan and I am here with Ian Robinson. Ian, what do you currently consider to be the best technical option for creating a service-oriented architecture? WS-* or REST?

I think it is always going to depend; we are always going to have heterogeneous environments within the enterprise. There are likely technologies that are already in place, applications that are already in place that use WS-*, and it is unlikely that we would want to replace those just to impose some kind of uniform solution. A lot of the stacks offer a kind of homogenous development environment. And if we are developing the internals of an application or the internals of a service we can certainly take advantage of a lot of those WS-* compliant applications and interfaces. I think once we are looking for tremendous reach and scalability, when we are looking to extend across organizational boundaries, then we might want to look at more RESTful solutions. We will have technologies at either end, technology stacks that have simple HTTP clients, we can take advantage of those, we are not having to worry so much about incompatibilities between different versions of a WS-* specification. So I think really being able to take advantage of some of the web's infrastructure, some of the scalability that is inherent in that infrastructure might guide us towards adopting a RESTful solution for those parts of our SOA.

2. One of the other questions which comes to mind is how do you view the notion of contracts within a REST scenario?

I think contracts, as we are used to them from the web services stack, aren't necessarily as applicable in a RESTful environment or a RESTful solution. Nonetheless contracts are there, are present in one form or another, and it's probably worth investigating those in detail. But first I will just talk a little bit about those web services contracts: WSDL and WS-Policy. Together they are typically said to comprise a web service contract. WSDL exposes the endpoints and the operations that can be performed at those endpoints. And WS-Policy asserts some of the quality-of-service characteristics that might be associated with that service.

WSDL in particular seems to encourage a remote-operation view of the world. It's very static, very upfront we are kind of committing very early to the way in which we want to consume a service. In a RESTful solution typically what we are trying to do is guide a client towards its goal. So the client makes a request of our service and we'll serve up some kind of representation, it might be a representation of an order for example. It may be that we want to progress that order through several states, ideally what we want to do inside that representation is advertise some of the possible transitions to the next stage or the next state in processing that order. So we're not necessarily having to advertise up front in some external contract what is that we can do with an order, the representation itself offers up several opportunities to manipulate that order. So this is that "hypermedia as the lever to application state" constraint within REST.

So really the contracts are being exposed gradually in a rather more dynamic fashion. We are still treating with the client over the course of several requests and responses. But we are enabling that client to make decisions on the fly as to where it wants to go next. One of the things that I think we really want to try and adhere to here is careful use of media types or MIME types. So a handful of good media types within their processing model, basically advertise what is a hyperlink, what are those hypermedia levers that are available within that particular media type.

So if we are building rather generic clients that can handle these particular media types, then they can begin to identify those possible state transitions, those possible on-the-fly elements of the contract and begin to act on them there and then. So we can start layering some application-specific intelligence on top of some very generic REST clients and we are using media types to guide those clients together with the particular representations that we are serving up. We are using those to guide those clients towards the successful completion of their goals. Does that make sense?

3. Yes, it does. So what are your thoughts on WADL, Web Application Descriptor Language?

We could use WADL. WADL effectively allows us to describe some of the operations that we could perform against any particular resource. We could use that in a WSDL-like way to provide some static upfront description of the service. Or we could actually use WADL to annotate some of those representations as they come back and say "Look here is today's contract for this order. Here are the things that you can do with this order today with this particular representation".

We have actually used WADL to annotate that representation and provide some kind of contract-like semantics on the fly. I think that is a preferable solution. Obviously we've talked about other media types, things like RDF has a rich processing model that allows us again to identify particular links and the semantics attached to those links so that we could interrogate a representation and then begin to progress it, in making further requests of the service.

4. How would you approach a large-scale company-wide SOA project?

An approach we have used successfully in the past at Thoughtworks, this is an approach I have used with a number of companies where they often have a very successful pedigree in mainframe application development and in other more recent kinds of application development as well, but they don't necessarily have any experience at SOA. Nonetheless there are very strong pressures for some significant change to take place, market conditions are changing, they are finding that their existing applications are difficult to change, expensive to change so all of this leads them to believe that they need some kind of SOA solution.

There's an obvious need there; they have a lot of experience in other kinds of application development; and they are aware that there's a lot of SOA experience out there but they don't necessarily posses it at this point in time. What do they do? How do they get started? What we are trying to do is provide some kind of accelerated route towards identifying and developing services, and then delivering them into the enterprise so as to deliver very real significant business value, value that meets that company's strategic goals. So typically what we do is present a very very simple map, ideally I would like to be able to draw that up for you, but it basically progresses from left to right and on the left hand side I just draw few boxes that represent some of those organizational units, their key goals, what is it they are trying to achieve, what's impeding them... In the middle of our map we diagram capabilities, and then over on the right-hand side we describe services and then specific technical implementations of those services.

And I say "What we are going to do together over the course, say, of a couple of months is we are going to begin populating this map with detail, we are going to start attaching very specific artifacts to parts of this map". So we are going to start trying to understand who some of the key stakeholders here are, what are their key goals, what are they trying to achieve, what's motivating them? And we are going to do that through a number of workshop exercises, things like that. And from that we are then going to start trying to identify some key or core capabilities that belong to your organization. What kind of capabilities or what abilities do you need to be able to furnish in order to meet some of those key goals? So again you want to draw those out as quickly as possible, doing just enough work to get some kind of sense of the overall scope of the engagement. Then we are going to start trying to assign those capabilities to services.

We say that services are hosts for one or more business capabilities. And then we can begin making some decisions about how we actually want to implement those services, what specific technologies, what architectural approaches do we want to take, in order to implement those services. So boiled down to its core, I talk about stories, capabilities, services and contracts. Stories I take from things like Behaviour-Driven Development, and using stories and the story format to describe a role -- "I want to achieve this, so that" -- and then I describe some kind of value attached to achieving that goal, so identifying a role, a goal and a value attached to that goal.

We use that kind of story format very often with people with strategic responsibilities at the beginning of an initiative. We'll ask them really to try and describe some of their key business goals, and the value that they attach to them. And then, as a complement to that, we try and derive what sort of capabilities are that that company possesses or needs to furnish in order to meet those goals. So what we are trying to create is a capability map, very simple, often this kind of hierarchical description, but it's a description really of what it is that the company does or what it is trying to achieve. And then that becomes the basis for a whole bunch of other conversations both with strategic stakeholders and some of the operational staff.

Which of these capabilities are core to your business? Which of them differentiate them, or differentiate you from your competitors? Which of them do you do well today, and which of them do you do badly? Are there any that you could outsource? And what are your kind of quality-of-service expectations around these capabilities? And we are not at this moment in time talking about how we implement them, but if you need to be able to source parts for second-hand cars, what's the turnaround time on that kind of expected provision of that service or that key capability? So we can ask all of these questions, we can begin to derive some kind of quality-of-service characteristics.

Then we are beginning to hone in on some of the key capabilities, the things that are very very significant, very important to this company, or things that they are not currently doing well but which nonetheless they ought to be doing better. We can then start to identify services, start assigning those capabilities to services. And then we can start making some decisions about how to actually build these services. Now often that takes place in an environment where there is a whole bunch of in-flight projects, so again as part of this very quick start way of approaching an SOA initiative, we start to try and map some of those key capabilities and perhaps some of the candidate services that we've identified to in-flight projects so we are beginning to create this shared understanding between several teams, several different groups of stakeholders, we are trying to bring them together within this very simple map, and help them understand the several dependencies between different project streams.

This is really an iterative exercise, so as I say we'll go through a number of workshops to identify some very high levels and key capabilities and we might immediately, following on from that, try to identify some candidate services and start trying to deliver very quickly some working software that helps satisfy some of those service behaviors. But we'll constantly be going back, engaging more stakeholders, drilling down and discussing more of those capabilities in detail. But again it's a conversation around what is it you are trying to achieve, why is it important to you, what kind of quality-of-service characteristics do you attach to these things? So this is the basis of the kind of conversation that can join up several different parties within the organization, some of those business stakeholders, some of those technical stakeholders. So the capabilities become this ubiquitous language and then the services and the service implementations are really implementation details. Now we can begin to share some of that with that group of stakeholders, but really I see the link as being really those conversations around the capabilities.

5. One of the questions that comes to mind is how does implementing a SOA change an organization which did not previously have one? How does it change the flow of work within an organization and capabilities?

I think one of the first effects really is a whole bunch of people who have not necessarily talked to one another too much actually coming together and creating a shared understanding of what it is they are trying to do, and why it's important, and doing it in some very simple practical terms. We are not using strictly technical language at this point in time. So we're creating this shared understanding, and often that's a real breath of fresh air for some of these organizations where they have been locked in very siloed efforts, there is a lot of repeated or duplicated effort across the organization. We are beginning to identify some of that and we're helping people bridge some of those gaps.

As we start actually delivering working software, we are also trying to encourage those teams to collaborate, to identify dependencies and responsibilities. This team over here might have some very real responsibilities to your team over there, how can we communicate those and how can we continually enforce those responsibilities. And one of the ways that we like to do that is to actually share tests. So your team could actually create a suite of tests that assert some of the expectations that you have of my service -- this is the way in which you want to interact with it.

These are the parts of the service that are important to you. And you give us those suites of tests, and we might incorporate them into our continuous integration environment, so we have this very practical programmatic asserted behavior. So we are actually getting these programmatic contracts being exchanged between teams. We are also trying to encourage teams to be more long-lived, so they live with a service from its inception through to its operation rather than a core team being dedicated solely to development and then handing over to some other support function.

Now again that's often quite a significant organizational change, it's not always easily accomplished and it's not always appropriate, but trying to encourage teams to have this long overall duty of care to the lifetime of the system and to take account of the several different parties that are going to be responsible for that system and thinking about that a little earlier in the development lifecycle. So again there are those kinds of very practical changes that we begin to see take place within an organization as people are coming together around these shared goals and as we are thinking of very practical ways of communicating, exchanging understanding and creating these programmatic contracts between teams.

6. One of the thoughts which comes to mind while listening to what you are saying is that a lot of what you are describing sounds a lot like an Agile implementation. Do you see a good SOA architecture within an organization and Agile being necessarily intertwined?

Yes, but I want to caveat that quite a bit. There are three terms that I'm quite wary of, particularly around SOA initiatives: those terms are Agile, integration and business process. So, I think it's a kind of open question even today: can we create organizational or enterprise agility using Agile project management or software delivery methods? So people often say: can we do Agile SOA? Sometimes that's a CIO is very skeptical, saying the rigors of SOA don't seem to go with this, seeming gung-ho attitude that Agile has, how can the two come together? And sometimes it's a kind of dogmatic Agile practitioner worrying that the kind of protracted exercises that seem necessary to SOA can't be made Agile and therefore they are almost questioning SOA's right to exist.

So can we do Agile SOA? To me that's the wrong question. Better to ask: what can we do to better meet an organization's key strategic goals? Whatever is appropriate is appropriate. Now as Agile practitioners, we've got a whole bunch of activities, practices, principles that we can bring to the table. Some of them are more appropriate than others, some of them have to be modified. Because we are dealing often with teams at a distance, projects working within different time streams, it's very difficult to coordinate all of those things. We can't always just pour our knowledge out onto a bunch of cards around a table, we are often dealing with a scale where that's not practical or possible.

Nonetheless, there are some key things that we take away from Agile, this desire to have insight, day-to-day or minute-to-minute insight into our systems, and the way they are behaving, desire to have very strict and close feedback loops. So I'm often looking for strategies and techniques and practices that can encourage those things and that's an attitude that I bring from Agile, but whether or not I am doing Agile SOA to me is an ecumenical matter, I am not overly interested whether or not I am doing Agile SOA. I said a couple of other terms that I am wary of as well, integration. So I often see organizations with some kind of integration group, or they are talking very often about doing integration.

Integration to me is a bit of a bad smell, it suggests that what you are trying to do after the fact is glue together a bunch of systems and fix up a bunch of bad decisions. So I prefer to talk about intrinsic interoperability, though I can barely say the word, and within a RESTful solution that is often leading to serendipitous reuse. But intrinsic interoperability, even worse the second time, over integration. Now I recognize that integration is absolutely necessary. There's a whole load of integration activity that takes place day-to-day and very often when we are building out a service a lot of the internal implementations of that service might be a bunch of integration activities. But let's not prize integration as something that we do successfully and it makes the world a better place.

And the last term that I am wary of, and again I recognize that it is necessary, but it's this term "business process". Business process to me, it seems an overly formal term, it's a term that is used by a bunch of specialists but it's not necessarily meaningful to those people on the ground who are actually achieving a company's goals day to day. There's a whole bunch of ad-hoc activities, collaborations that take place. If you are of a particular strategic mind, you might try and divine business processes or overlay this formality on top of them. But if we talk too quickly about business processes we end up talking about these rather fragile assemblies or sequences or workflows. And we say "Well this is the way the world is today and this is the way it ought to be".

In fact a business process again is often an implementation detail of one of those capabilities. Today we implement the capability to source second-hand parts for your car, some hurried staff in the call center have to go through a filing cabinet, that's the business process today. Tomorrow we might automate it, but they are implementation details. So again I recognize that it is a very important term, but if I am looking for an Agile SOA solution that is oriented around integration and business processes, I think I am slightly off track. If I am trying to deliver to some of those significant goals, providing solutions that are capable of evolving, where we've got rapid feedback and minute to minute insight into the behaviors of our system, that's good. If we are doing integration inside of our services that's absolutely necessary integration but protecting all the other consumers of that service from that kind of messy detail, that's good. And if we are talking about business processes as a pure implementation detail of some of those key capabilities, again that's fine.

7. Can you talk about the current state of tool support for developers who want to get going with REST-based web services and frameworks and tools and best practices?

Ok, so the state of tool support for doing RESTful development. Well, the base tools are there in most development languages - HTTP clients and ways of hosting solutions so they are listening to an HTTP endpoint as well... Let me have a think about that. I am not overly familiar with a wide variety of tools, and I would say I suppose there isn't terrific tool support in a way that there is with the WS-* stack. So typically as a developer what we want when we are developing a web service, is some WSDL that we can then use to autogenerate some kind of proxy.

If some of the representations that we are serving up in a RESTful manner conform to some XML schema, then it's likely we have got some kind of automated tool support for producing those strongly-typed representations in our own language for serializing and deserializing. I don't think that's absolutely necessary, I much prefer to, again if we are using XML, to XPath out the parts of the message that I am really really interested in and discard all the rest. And I'd probably adopt that approach even when using the web services stack. So again my preference is not to autogenerate proxies and clients against a particular version of the schema, but instead just to parse out the bits of the message that I am really interested in, and I would adopt exactly the same approach in a RESTful solution.

What I have done a little work on recently is a very simple DSL for expressing a client's expectations with regard to a particular message or representation. So that DSL it looks a little like YAML basically a bunch of nested terms together with a type, I expect this to be a string, I expect this to be an integer. But it is very concise but then from that I can generate a whole bunch of things, I can generate a bunch of XPath assertions so that I can validate incoming messages if I want or I can validate them on the way out. I can generate serializers and deserializers that are dedicated to satisfying my expectations with regard to the kind of messages that you are producing. I can also use it to generate a graph that, with regard to this particular schema, this graph begins to describe some of the expectations that different clients have of that schema.

So I am building up effectively this kind of social networks for contracts. So all these different artifacts we can generate off the top of something produced with this DSL. It's something that I am playing with at the moment, but that seems to me to be a way of being able to express my expectations of a message and then create type classes that are really dedicated to those expectations, towards servicing those expectations. So I've kind of got a bit off track in terms of your question around REST tool support. In developing some solutions recently with Atom and AtomPub, what I've really wanted to ensure is that the protocol and the way in which those clients are interacting with those feeds is being adhered to, so I want to create a whole bunch of unit tests around the service that is generating those feeds.

And I want to be able to assert that specific HTTP headers are coming back, certain response codes in response to a particular stimulus. What I found, and I was doing this on the .Net framework, what I found was that you can very quickly get into that HTTP context, but for every test, what you are having to do is actually instantiate a service over HTTP and communicate with it. So what I have done is just create a simple wrapper around that HTTP context, it's an interface that I own, and then I can mock it out and obviously set expectations with regard to that mock. And then I've got a separate bunch of tests that just assert that specific implementations of that interface actually delegate to the .Net framework. One of the things that we want to be doing is testing the protocol; like I said, that's in terms of status codes, headers, that kind of stuff.

8. How can you achieve reuse, and how can a provider foresee the needs of its clients?

Going back to something that Martin said in the keynote this morning, our experience is that reuse often happens after the fact. Let's do one thing well -- we understand the specific context in which this particularly piece of functionality is to work, let's deliver to that. Then we begin to identify opportunities for reuse. If we have a suite of unit tests then we have a bedrock of asserted behavior that will allow us to evolve a piece of software towards a more generic solution. So as new use cases come along, we are identifying those opportunities for reuse, we can evolve our software quite rapidly and that suite of tests helps ensure that our existing obligations are being satisfied.

What we have done at Thoughtworks with a number of clients on a number of projects is then we'll extend this, and it's something that I talked about earlier in fact, whereby, if we are talking about reusing aspects of services, parts of services, then we'll often have client or consumer teams give us a suite of tests that describe their expectations with regard to our service or, if we are earlier in the software or the service development lifecycle, then together these teams will help establish obligations and expectations. You need this of me, you need to see this happen, how can we describe that as a contract between the two of us, and how can we then turn that contract into perhaps a suite of tests, perhaps again XPath or Schematron, that say "you send me this message or if I send you this message this is the kind of response that I expect".

And these are the parts of the response that I am particularly interested in. So, when you give me that suite of tests you are basically communicating to me your expectations and then it may be that another client or consumer comes along and gives me a similar suite of tests, and gradually I'm building up my overall aggregate set of obligations, so I'm beginning to learn. This often takes place within a controlled environment, within the enterprise, it's not necessarily an Internet-scale solution. But I'm beginning to understand what my obligations are with respect to several different parties and then if I want to evolve my service, well I'm free to do so just as long as I don't break any of those expectations.

On your side what you're promising is, you know, you've advertised an enormous schema - actually we are interested in these five different fields, what we're promising is that we are only going to consume those five fields and we are going to throw anything else away. So you're free to change everything else as long as you continue to provide those five fields, so the promise on your side is you are not importing or making use of stuff that you are not telling me about. And then as a service provider I'm quite free to change my schema to evolve it as long as I don't break any of those existing obligations and in fact I can make a change that is ostensibly a breaking change but as long as it is not actually breaking any of my extant clients, who cares? So the versioning... There's definitely versioning taking place here, but it is often at a slower pace.

I don't necessarily need to version if I know that I'm going to continue to satisfy all those existing clients. It's when I identify a real breaking change and I need to communicate that to you, we need to be able to identify some way of moving forwards. That might be that I provide an alternative implementation that supports you for the next 6 months, it might be that you have to begin to make some changes now.

9. You had mentioned versioning. What do you consider some of the best approaches with regards to versioning?

I think it's firstly about providing a platform for evolution. So with XML schema we can provide for extensibility points, we can design schemas with extensibility in mind. That's often quite cumbersome, and the messages as they've evolved over time actually begin to look rather awkward and are not necessarily as expressive as they might be. I've talked a little around what we're calling consumer-driven contracts and the fact that they help me understand when a change is really a breaking change and when what's a seeming breaking change actually doesn't really disturb the universe at all. So, there are very real demands for a versioning strategy within an organization.

You know these things can often be very long lived, and we've seen mainframe applications that have lived for twenty years or more, it will be wonderful if the kinds of solutions that we are producing today could have a similar lifetime. It's almost inevitable that they are going to change and therefore we do need to start thinking about those versioning strategies. I don't think there is actually a great versioning story in a lot of tool sets today and in a lot of the frameworks and I think it is a problem that is beginning to make itself felt, and I think a lot of those technology stacks and a lot of solutions and the frameworks are beginning to address that. But I try and take a very cautious approach, basically having consumers only consume what's absolutely important to them, discard the rest, have them try to communicate some of those expectations to a provider, and that helps the provider understand when they are free to change, but at some point we do need to version, and that's the point where we might have to take advantage of some of those extensibility points that we have provided for, it might be that we have to provide a wholly new interface.

10. What do you see as the future of REST-based web services in enterprise SOA development?

I think we are learning today that a lot of the enterprise solutions that we have built in the past are very much confined to the enterprise, and we have often abused or completely disregarded some of the benefits that things such as HTTP and the larger web infrastructure have to offer us. We are also discovering today that a lot of the value that we want to generate within an organization is dependant upon its interactions and its collaborations with other organizations. So far more communication across organizational boundaries. Parts of the web services stack inhibit that kind of cross-organizational growth. We have a proliferation of specifications and often for a particular specification there are several different versions.

We are finding it increasingly difficult to get that kind of intrinsic interoperability across organizational boundaries using the web services stack. RESTful solutions can help us extend our reach in this regard. We are taking advantage of a constrained interface, but we are beginning to surface and describe a rich pool of resources and we are helping identify each of those resources and make them available to our clients and to other organizations. And we are helping guide those clients towards successful, the successful conclusion of their goals. So we talked about that earlier in terms of serving up representations that help a client achieve its goals and we are beginning to advertise what the next step in the process is.

Now, I think that we can learn from that even if we want to import some of those lessons into the way in which we are building solutions on top of the web services stack. Identifying resources in and of itself is a very useful exercise, so adopting kind of resource oriented thinking often help us identify things which are significant to a company, which generate value on behalf of a company, give them a name. Often those things are otherwise buried away in some implementation detail - we're beginning to surface them, give them names, make them addressable. The idea that processes are not defined once and for all, that they might gradually evolve over the course of a long-lived conversation, interaction across organizational boundaries.

Again, how can we guide clients, how can we advertise what is possible to do today, we might be advertising something completely different tomorrow, we might be introducing for example some kind of advertising campaign in the midst of the ordinary process and if a client can recognize those additional elements of that process, that we're advertising on the fly, they might be able to take advantage of that. But clients that don't recognize it can still meet their core goal of generating an order. So, I think we are seeing solutions today that are emerging that are beginning to take advantage of some of this thinking, beginning to introduce some RESTful ideas across a broader range of solutions and that is the kind of influence that I would like to have in the next few years.