Tracking change and innovation in the enterprise software development community
Scaling Scrum
in the enterprise?
ScrumWorks Pro
can help.
Start a trial today
Free Book Offer
SOA Demystified
$40 Value
QCon London 2010!
March 10-12, 100
Speakers, 80 sessions,
"THE" team leads and,
architects conference.
Posted by Sadek Drobi on Oct 10, 2007
“For some reason, we have invented and are following religions […].” This is how James O. Coplien describes today’s industry which, he believes, is based on buzzwords and checklists rather than on thinking, inspection and efforts to find solutions that would be the most appropriate and the most cost-effective for a given project:
How much thought did you put into the tradeoffs of the last technique you brought into your organization: Ajax, TDD, On-Site Customer, or other buzzwords? Did you research its track record? Or did you go to the buzzword yellow pages?
James uses the example of the debate around TDD which has occurred in the blog space in the wake of Jaoo 2007 conference. When the discussion focuses on what approach to testing is the best (i.e. TDD or acceptance testing), it essentially addresses the wrong question. It obfuscates the real objective which is “to deliver what the customer wanted while optimizing quality” and to create value for the client. Testing is what we have on today’s checklist for quality but it is far from covering all the issues that actually affect quality:
Quality suffers if you misunderstand what the customer wanted, or if the code has internal interactions that are difficult to foresee, or if local code violates array boundaries or uses untethered pointers. It also suffers if the interface makes it possible for the user to commit errors, or if it takes too many keystrokes to enter a small amount of common information.
Focusing on quality requires considering every weapon in the arsenal:
That means using Use Cases (which are an Agile way of engaging customers […]) instead of XP-style user stories (so you understand feature interactions up-front), doing pair programming or desk-check code inspections, doing design by contract, driving the process with usability experts and doing good usability testing, and a host of other things.
Coplien emphasizes that it is impossible to do everything and that it is “a matter of cost effectiveness”. But being focused on techniques and driven by buzzwords we get stuck in practices that might not be optimal. The use of some techniques and methodologies, e.g. TDD, has become “a religious issue”, according to James:
We are told that you have to do TDD to be a professional […] and yet are given no reason to believe this, no substantiation, and no evidence. Just believe.
He argues, for instance, that “integration and system testing have long been demonstrated to be the least efficient way of finding bugs”, that TDD “deteriorates the architecture”, and that “acceptance testing is orders of magnitude less efficient than good old-fashioned code inspections, extremely expensive, and comes too late to keep the design clean.” But he stresses how difficult it is to engage a challenging discussion on such “religious” topics because any criticism is met with a lot of emotion.
People have tied their Agile success to their introduction of TDD practices into their work. It's the one thing they can do as individuals rather than as an enterprise.
Coplien advocates for focusing on quality rather than testing and, more generally speaking, on thinking rather than checklists. He highlights however that “sorting these things out requires a system view”, which often times lacks in today’s industry. He believes that “New Age Agile movements consistently move away from” systems engineering. According to Coplien, “one root of the problem lies in modern education, which has increased its focus on technique and reduced its focus on thinking.” Today’s students “fewer and fewer understand software history” and “have stronger memories […]on exactly when to use --v and when to use v-- than they recall anything about logical design.”
Close links which exist between academia and industry are also a part of the problem. Universities tend to adapt curricula to industry’s needs in order to facilitate students’ placement. Hence, it becomes difficult for academics to challenge predominate views, probably “as difficult as it was for Newton” to prevail “with new insights on the working of the universe”:
In fighting the broad misunderstandings that industry has chosen to adopt from selected interpretations of Agile, SOA, and programming language buzzwords, we face no less a religion, and no less threat to our cozy industry-financed academic positions, than he did.
Learning to be a Good Product Owner: Foundation Skills
Understand and Align BPM, SOA, and Lean Six Sigma: 43 page IBM Redguide
VersionOne is recognized by Agile practitioners as the leader in Agile project management tools. Companies such as Adobe, BBC, CNN, Dow, HP, IBM, Sony and 3M have turned to VersionOne to help deliver greater value to their customers.
I'm treading on dangerous ground here, but I'm not sure Jim Coplien should be taken as an authority on TDD or Agile in general. On page 1 of his book, Organizational Patterns of Agile Software Development:
O.K., we'll be frank: we chose "Agile" for the title out of marketing concerns. It seems to be the current term of choice for the kinds of things we describe in this book. It is a term that rolls off the tongue more easily than other clever names that clamor for your attention on today's bookshelves. ...
In many of his writings, I have to question whether he has ever experienced TDD and 'suspended his disbelief' before pronouncing his verdict.
Moreover,
“integration and system testing have long been demonstrated to be the least efficient way of finding bugs”
Where did this come from? In fact, system testing is the probably the only other topic - other than code inspections - that has been thoroughly documented and tested with experiments and have been shown to improve code quality. If you have the appetite for some academic reading:
A quick search on scholar.google.com will give you months of reading. You will find hundreds of papers that have been written on the effectiveness of software testing and their strategies.
So, with all due respect, I have to think that Mr. Coplien's arguments feel biased and, in some instances, are unfounded.
O.K., we'll be frank: we chose "Agile" for the title out of marketing concerns. It seems to be the current term of choice for the kinds of things we describe in this book. It is a term that rolls off the tongue more easily than other clever names that clamor for your attention on today's bookshelves.
I'll give him credit for candor, though it would have been better to leave "Agile" out of his title. Agile has become primarily a marketing platform.
I agree with his points about not having data to back up assertions made in support of agile (or the latest buzzword).
I read James' posts regularly and normally enjoy them, but I thought this hid a good message with an unfortunate dig at TDD, checklists and New Agile (whatever that is..).
Thinking test-first is, in all situations I can think of, better than test-last. Is it a substitute for holostic quality assurance? Of course not, it's just another weapon in the arsenal. Are checklists a substitute for thinking? Again, no. And so on.
His underlying theme, which is that IT in many organisations (although I'm not keen on vast generalisations) is missing something by not understanding software history, getting hooked on bandwagons, buzzwords and one-size-fits-all religions is IMO on the money. And sure, I've seen plenty of bad examples of TDD, and an overdependence on 'process' over common sense, but that don't make 'em all bad. It just means common sense ain't that common.
Part of the problem is that the term "Agile" is overloaded. But in my view this is not the type of overloaded we're most familiar with, nor is it a wise step to let the current movement have sway.
I run into this all the time: When I say, "I'm making my system agile", or I'm using an "agile technique", I'm using the dictionary definition of the word, and there's no way that I'm going to allow book publishers and process theorists to steal a very useful English word from my lexicon, no matter how great their process is. When we say "agile" it means exactly what the dictionary says it means, as far as I'm concerned.
This differs from "Scrum" in that the terminology, when applied to a technique or process, is just not easy, nor perhaps even possible, to confuse the usage with rugby football, where I think its more frequently used.
So it's perfectly acceptable for Jim, or anyone, to use the term agile, and in a sense, protect a term from misuse by marketers and theorists. Perhaps the theorists can use the Special Theory of Relativity example, if they want to specialize the Agile term, and say something more identifiable, such as "Agile Process of Test Driven Design". Otherwise, they just don't have a claim. Unless they want to be like one of those more unethical organizations that try to and sometimes succeed at trademarking a common word for an overly broad purpose.
This might all seem like terminology and shibboleths, but the cream of the crop of today is the stinking cheese of tomorrow. We all watched and learned a thing here and there from the XP zealots, taking their advice under advisement, and it was a good thing, but where is all that today? The Agile TDD process is going to morph and change, and eventually be replaced by some new religious dogma, and hopefully along the path we all learned something. But the English language will still have a great little word that still just as useful for describing whatever it is we feel like :-)
I think even more interesting is applying Jim's ideas to the RESTefarians who seem intent on their way being the only way and cutting down anyone who dares question GET, PUT, POST and DELETE as being the one true integration technique. Now there's great little example of turning good ideas into bad blood, and a great way to start arguments whenever I feel like it ;D
The norm for reasoning about adopting ideas is a pyramid. The majority forming the base are 'religious' thinkers. What matters is who supports the idea. We will let those people do our thinking for us, we trust them. The next layer in the pyramid is the political layer. We listen to the discussion, we think we understand it, we are convinced. Again, the people who support the idea are a significant consideration, but we're prepared to accept good ideas from other sources if people can convince us, and we're prepared to accept that or 'guru' might not be right on the ball all the time. Above these people are the scientific and finally the mathematic thinkers. Even the Scientific thinkers cannot check and verify everything; they build up a network of people whose work has so far proved to be correct, and whose endorsement of other people's work can be trusted.
This is to be expected. We cannot change this reality much. We need to learn how to get this reality to work in our favour. Unfortunately, politicians and con-men are well aware of this model; we have competition. As a cynic, I'd have to add that Politicians ultimately control the education that can alter the shape of the pyramid, and they like having some people they can fool all the time.
I'll give him credit for candor, though it would have been better to leave "Agile" out of his title. Agile has become primarily a marketing platform.
I agree with his points about not having data to back up assertions made in support of agile (or the latest buzzword).
First, you quote a bit out of context. Go back and read the ensuing paragraphs. Consider it a homework assignment. I trust you'll report your findings back here.
Second, the work has strong stature as being a major grounding of the Agile discipline as it emerged with an identity. Mike Beedle, noted Scrum author, calls this book perhaps the first documentation that ever existed on true Agile development. No one, to my knowledge, had done so before. (Not Scrum, which started in 1993, nor XP which started much later. etc.) Kent Beck, in his biographical summary, acknowledges the role that these patterns (first published in 1993) played in shaping XP. Jeff Sutherland, with whom I work regularly, points to the original Borland paper behind this work as the origin of Scrums.
So I hope I've substantiated my credentials; if this doesn't do it, I don't know what can.
To establish one's credibility through uninformed character assasination is the tool of the ignorant and the weak.
And I hate it when people can't even get their history straight.
As regards TDD, I recently sent a mail to Bas Vodde detailing my involvement with TDD and formalized design support dating back to the 1980s. (Remember, TDD goes back to the 1960s.) We used it successfully in various Bell Labs projects built in C using the procedural paradigm--however, we noted many problems even there, particularly those related to the cost of changing tests (Cause-Effect charts) that were written at an overly detailed level, so we resorted to chunkier development units. My place in that world was to develop some of the tools that supported TDD. What worked was to build-and-test bottom-up--a natural pattern when using the procedural paradigm. When we moved on to develop and apply the toolset for test-driven development of object-oriented programming in the 1980s, it fell apart. The tools we had then make xUnit look like a toy (there is a small bit of published work still around from those days.
I also have a long history working together with Miron Abramovici, one of the pioneers in design for testability in hardware. At my last job, in a company founded by Miron, we were building a hardware design infrastructure based on introducing test points into the circuit spec before taking the circuit into elaboration and layout. Miron and I led the way to guide the programmers to delineate their interfaces using assertions tracible to business value. I think I've had enough experience with TDD in enough different settings over 30 years to be justified in offering a perspective on it.
Now, let's move on to discussing the real issues instead of trying to take the cheap way out of difficult discussions.
For practical reasons, I'll not be back to this 'blog for a while, simply out of need to manage my time.
If you want to have an honest discussion, please come visit at my 'blot on Artima.
But, before we do that: What are your credentials, and which of your writings can you substantiate?
I must have touched a nerve made raw somewhere else.
Suggesting that it would have been better to leave Agile out of the title is hardly character assination. My comment was based on what you wrote in the forward to your book about the need to put Agile in the title for marketing purposes. Hence my comment about Agile being a marketing platform.
BTW, I started a thread on Agile becoming primarily a marketing platform several months ago on the Agile Management Yahoo Group, so that is not a comment aimed at you. Your comments in your forward do confirm that though.
Brian Marick takes us through a quick tour of the most important values and challenges to adopting Agile successfully (they aren't the typical challenges and values we hear in the community).
The line between development and architecture is tricky. Does it exist at all? Is an ivory tower actually needed? There's a balance in the middle, but how do you move from developer to architect?
The word 'authority' sometimes produces an allergic response in hard-line agilists. Freedom and authority – both are bad if misused and both are good if used in right spirit for a noble cause.
"Getting Started with Grails" brings you up to speed on this modern web framework. Companies as varied as LinkedIn, Wired, and Taco Bell are all using Grails. Are you ready to get started as well?
Those familiar with only ITIL V2 often scoff at the thought that ITIL could serve as a governance framework for SOA. With ITIL V3, the focus of the framework shifted towards service-orientation.
SpringSource CTO Adrian Colyer discusses AspectJ, SpringSource's dm Server and tc Server products, OSGi and Scrum.
Heroku's Adam Wiggins talks about Rails, Background Jobs, Add-Ons, Ruby, and how Heroku manages to work around Ruby's inefficiencies using Erlang and other languages.
For Grady Booch the foundation of a good architecture is patterns, SOA being just one of many patterns. In this Second Life presentation, Booch attempts to bring more clarity on what architecture is.
7 comments
Watch Thread Reply