Tracking change and innovation in the enterprise software development community
QCon London 2010!
March 10-12, 100
Speakers, 80 sessions,
"THE" team leads and,
architects conference.
Scaling Scrum
in the enterprise?
ScrumWorks Pro
can help.
Start a trial today
Free Book Offer
SOA Demystified
$40 Value
Recorded at:
Presented by
J.B. Rainsberger
on
Sep 10, 2009
Download:
Learning to be a Good Product Owner: Foundation Skills
Design in Agile, Software Craftsmanship, Product Development, Testing @ QCon London in March
VersionOne is recognized by Agile practitioners as the leader in Agile project management tools. Companies such as Adobe, BBC, CNN, Dow, HP, IBM, Sony and 3M have turned to VersionOne to help deliver greater value to their customers.
video doesn't work for me
Hello Scott,
i just tested this both in Firefox and IE and it seems to be working fine. Could you please try again?
thanks
Diana (InfoQ)
A Mars rover mission failed because of a lack of integration tests. The parachute subsystem was successfully tested. The subsystem that detaches the parachute after the landing was also successfully (but independently) tested.
On Mars when the actual parachute successfully opened the deceleration "jerked" the lander, then the detachment subsystem interpreted the jerking as a landing and successfully (but prematurely) detached the parachute. Oops.
Integration tests may be costly but they are necessary. Sometimes there aint no substitute for end-to-end testing. These testing concepts were developed 50+ years ago in the fields of quality control and industrial engineering. I wish more "Agile" community was aware of this existing body of knowledge.
Sounds like a great subject of an article or 2 or 3 or.... Amr
I believe that J.B. was focusing on "software" integration tests rather than "system" integration tests.
Hi Joe Rainsberger, (I hope you see this message)
First off, this is one of the best presentations I've seen in a long time - very inspiring.
I was thinking about what you said about contract tests. In the system you describe every interaction has a mocked call to an interface if I understand it correctly. Each of these interactions must be verified to be supported on the interface for the contract to be fulfilled. If we could in some way describe the mock expectation as a data structure that also could be used by the contract test, wouldn't we be home safe? Then we should just make an expectation on the mock object using the data structure and use the same data structure in our contract test to make the assertion. In a way our expectations on the mock is the contract on the interface.
Sorry if I'm just totally wrong.
Kind regards Sune Simonsen (Jayway)
Unit-tested modules can still misbehave when they are combined. Integration testing ensures that modules operate correctly when they are combined, and finds unexpected dependencies or interaction among modules. JR's presentation correctly notes that exhaustive combinatorial testing is too expensive for integration testing, but that doesn't mean it can be ignored. Huge numbers of input combinations can be avoided by statistically testing based on usage models.
To be fair, JB is only claiming to establsih "basic correctness". So, for example with the method that returns a collection, he proposes "0, 1, many, lots". It may be that you also need to add the returns-something-that-will-cause-a-jerk. His scheme isn't talking about how you choose the good set of tests out of all possible tests.
John D.
If a test fails I use this new-fangled invention called a "debugger". It allows me to look at a running program and quickly determine why a test failed.
I can't understand why the so-called "Agile Comunitity" has such a hard time with this concept. Perhaps you should hire some college kids to teach you about debuggers in your bootcamps.
I had the following idea after watching that presentation:
Instead of creating the mocks and setting the expectations in the tests of a client class, move all the mocking code to static helper methods in the contract test class. For example the contract test class would have a method 'mockAnEmptyRepository()' and whenever some other test needs to collaborate with an empty repository, they would create the mock using 'RepositoryContract.mockAnEmptyRepository()'.
The goal of this approach would be that when the contracts and the mocks are in the same class (or at least very close to each other), it will be easier for the programmer to check that all the mocks obey the contracts.
Still another idea that I had:
In the contract tests, wrap the object under test into a proxy which implements the same interface as is being tested. When the contract tests are executed, the proxy will collect data about the method parameters and return values. Then this same data would be used to automatically verify that given the same initial state* and parameters, the mocks will return the same return values.
* By initial state I mean for example, that in the contract test class there is an abstract method 'createAnEmptyRepository()' which creates the object under test. The same contract test also has a similarly named method 'mockAnEmptyRepository()'. Here the "initial state" is "an empty repository". The analyzer will group those two together, and check that the object returned by 'mockAnEmptyRepository()' behaves the same way as the one returned by 'createAnEmptyRepository()' does.
Because of using TDD, we need a debugger so seldom, that our debugger usage skills have deteriorated. ;)
TDD (and specifically test first programming) is not particularly effective at testing certain types of software. Some examples are GUIs and multithreaded applications, both of which are quite common. I'd claim that TDD has deteriorated your software development skills in general by causing you to believe the myth that you don't need to know how to use a debugger.
1. TDD is primarily a design technique. Thinking of it as a testing technique gives a very narrow view of testing.
2. GUIs are best implemented so, that their logic is decoupled from their presentation. The logic can be driven with TDD, after which the presentation will be mostly declarative and simple glue code. The presentation will anyways need to be tested manually, to make sure that it looks right.
3. Multithreading is best when avoided. Design the application so that the need for multithreaded code is restricted to only a few classes (for example using message-passing).
4. I did not claim that there is no need to know how to use a debugger. I said that there is very rarely need for them (maybe once a month), because with TDD you find the cause for a failure much faster - if you changed one line and it caused the tests to fail, the probability is very high that the problem is on that one line of code.
I agree that appropriate software structuring can increase the effectiveness of testing. My point was that even with that higher effectiveness there are common cases where TDD cannot ensure the system is operating correctly. Multithreaded applications are common and cannot simply be avoided in most cases. Even message passing techniques alone do not imply the lack of shared state between multiple threads.
To slightly rephrase my original statement for clarification purposes, I was referring to the "myth that you don't need to know how to use a debugger [well]" because you are using TDD.
I like agile (even most Agile) techniques and I've been using TDD for almost 10 years. I seldom, if ever, use a debugger to diagnose a problem in my unit tested code. However, I use a debugger for other purposes and I believe it's important to not allow those skills to deteriorate.
Would it be possible to let the data structure that Sune mentions simply be the interface marked up with 'design by contract' annotations (pre, post, and invariant conditions)? Then, perhaps some unit testing frame work could generate the proxy that Esko mentions (I was thinking it might be an AOP aspect). The proxy would assert that both the consumer and supplier are abiding by the contract.
I enjoyed JB's presentation.
Indeed, J.B. does NOT claim integration tests are totally evil (as the title incorrectly implies), but rather that much of what is commonly "checked" (aka tested) via integration tests would be better verified with more isolated micro-(aka unit-)tests.
Cheers
MB
Frank,
I think what good TDD allows me to do is (for the grand majority of my time) ignore usage of a debugger to determine anything about the code I'm now developing.
Do I still need to fire it up (and be effective with it) to diagnose bugs sometimes - absolutely.
Cheers
MB
I suppose you've never actually come across a case where the system only fails on the production environment? Im sure your administrators let you point this new fangled invention of yours to this environment
The presenter really knows his business, explained clearly and convincingly the practical differences between focused tests and broader integration tests. Would have liked to hear more on the challenges of using test doubles in what are sometimes called white box testing. The mock code can end up mirroring the real code to closely, causing brittleness in the tests during maintenance. I've found the key to mocking is not over-specifying the expectations.
Well, taking shameless advantage of this off topic, I would like to report that roughly at 32 minutes of this talk it stops. If I try to play again from the buffer, it will also stop when I pull the bar beyond 32 minutes. This happens not only with this video, but many others, for some weeks now. I'm watching from Brazil with Firefox 3.5.30729.
Video stops for me at ~49:30. Replay stopped at same spot.
Hi. Very nice presentation, learned from it and also saw familiar things.
One question about Contract Tests though; it is said earlier that you should not test a platform. I assume this also means you should not test a Webservice (client code) you generate (lets say from a WSDL) your client code from. (makes sense)
Instead , the Contract Tests should test the interface of the service that you define (that will use the webservice eventually).
Do I understand this correctly?
Ie, to clarify, lets say I have a Webservice, that has an implementation using SOAP. I have my own service (interface) that will answer questions for my code / features using my service. I would end up with:
MyCode <-> IMyService (with implementation MyConcreteService using IWebservice->SOAPWebservice)
The Contract Test would test only IMyService (and has a test class for every implementation of that interface) as it is unwise (and especially slow) to test the actual webservice.
Correct?</->
I agree that we must mock classes dependencies and also do real Unit Tests.
I also think we need to automate some (not all, impossible, but some) of the functional tests (customer tests, end-to-end tests)in order to quickly see if some functional behavior breaks when some change is done on our code, even UT is passing.
An example is if you want to perform a re-factory on your code, probably you will lose UTs and you need an automated way that test the entire behavior from the interfaces signatures. What should be a good practice for this scenario?
Regards,
Filipe
"One question about Contract Tests though; it is said earlier that you should not test a platform. I assume this also means you should not test a Webservice (client code) you generate (lets say from a WSDL) your client code from. (makes sense)"
I don't generate web services from WSDL often enough to have formed an opinion on that specific case. I guess I would either have to trust the generator or not trust the generator, then act accordingly. If I consume the WSDL, then I would probably writing Learning Tests to document how to use that web service correctly, rather than writing tests to prove its basic correctness
Looking at your specific example, I would use Learning Tests to discover or confirm the behavior of SOAPWebService, then use that information to implement MyConcreteService correctly by stubbing/mocking methods on IWebService. The more I trusted SOAPWebService, or the product that generated it, the less I'd worry about writing exhaustive Learning Tests for it.
I certainly wouldn't test the web service client generator which, in this case, represents the platform.
me, too.
who can fix this?
"I also think we need to automate some (not all, impossible, but some) of the functional tests (customer tests, end-to-end tests)in order to quickly see if some functional behavior breaks when some change is done on our code, even UT is passing."
You equate functional test with customer tests with end-to-end tests. I use them differently. Everything I've said about integration tests apply to tests programmers write to gain confidence in the correctness of their code.
I usually automate all the routine customer-level checks in a system. Many of those end up as end-to-end tests, but in a highly modular system we can easily check a lot of business value without resorting to end-to-end tests.
"An example is if you want to perform a re-factory on your code, probably you will lose UTs and you need an automated way that test the entire behavior from the interfaces signatures. What should be a good practice for this scenario?"
I use a combination of collaboration tests and contract tests. To read more about those, go to thecodewhisperer.com.
Hi, J.B.. Great inspiring presentation. Congratulations!
There's a thing about contract tests that I think can be misleading.. Can you really rely on them to integrate things? When you a run a integration test, the method's client can get an outcome that you didn't test on your collaboration tests. Worse, you're mock can return a outcome that is possible (contracts test won't complain) but simply never happens in a real scenario.
thx, congrats again!
We still need to match stubs/mocks in collaboration tests to assertions in contract tests, but at least we have a systematic way to match them, rather than relying on people to do their best.
Brian Marick takes us through a quick tour of the most important values and challenges to adopting Agile successfully (they aren't the typical challenges and values we hear in the community).
The line between development and architecture is tricky. Does it exist at all? Is an ivory tower actually needed? There's a balance in the middle, but how do you move from developer to architect?
The word 'authority' sometimes produces an allergic response in hard-line agilists. Freedom and authority – both are bad if misused and both are good if used in right spirit for a noble cause.
"Getting Started with Grails" brings you up to speed on this modern web framework. Companies as varied as LinkedIn, Wired, and Taco Bell are all using Grails. Are you ready to get started as well?
Those familiar with only ITIL V2 often scoff at the thought that ITIL could serve as a governance framework for SOA. With ITIL V3, the focus of the framework shifted towards service-orientation.
SpringSource CTO Adrian Colyer discusses AspectJ, SpringSource's dm Server and tc Server products, OSGi and Scrum.
Heroku's Adam Wiggins talks about Rails, Background Jobs, Add-Ons, Ruby, and how Heroku manages to work around Ruby's inefficiencies using Erlang and other languages.
For Grady Booch the foundation of a good architecture is patterns, SOA being just one of many patterns. In this Second Life presentation, Booch attempts to bring more clarity on what architecture is.
28 comments
Watch Thread Reply