It has been more than 20 years since Mythical Man-Month, author Fred Brooks, published the article "No Silver Bullet: Essence and Accidents of Software Engineering". In his original paper, Fred Brooks argues that there is no innovation in software development that would achieve an order of magnitude increase in productivity (the silver bullet) in the ensuing ten years. Brooks identifies two categories of complexity in software development: the essence and the accident. The essential complexity of software development is related to the specification, design (mapping specification to software), and testing (that the design properly meets business needs). The accidental complexity refers to the difficulties related to implementation (languages, runtime, tools, and programming techniques). In his article, Brooks explained how the various innovations that attempted to address accidental complexity (at the time) were not silver bullets, and concluded that the only things that may yield close to an order of magnitude productivity improvement are those that address essential complexity, such as improvements in requirements gathering, rapid prototyping, and cultivating good design skills.
At the OOPSLA conference last year (2007), a retrospective discussion panel on No Silver Bullet was held including Fred Brooks himself, Martin Fowler (who later surprised the audience appearing as a werewolf), Ricardo Lopez, Aki Namioka, Linda Northrop, David Lorge Parnas, Dave Thomas, and Steven Fraser as panel impresario. The panel explored in hind-sight Brooks’ premise that there is no silver bullet in software development. InfoQ was present at the panel and has since lobbied heavily to bring this important discussion to the community. The following is an edited summary, published with permission, including key excerpts from the panel.
Steven Fraser opened the panel by asking those present if they read the article when it first came out 20 years ago. Three quarters of those present raised their hands. Steven told the audience that “I remember that it came out on the day of my doctoral defense. My thesis supervisor said it was a good thing that I didn’t say anything that disagreed with Fred.” Steven continued by presenting each of the panelists and expressing his sympathy for Ricardo and his family which was evacuated due to the tremendous wildfires which hit California at that time. Then Steven offered each panelist a few minutes to introduce their views on silver bullets in the last 20 years. Fred Brooks began with a recap of his initial idea:
I postulate that the difficulties with building software can be divided, following Aristotle, into the essence, which is the conceptual structure of the software itself, quite apart from any realization, and the accidents, (using Aristotle’s terms—you might prefer incidences,) the difficulties that are not occasioned by the conceptual structure, but are occasioned by one or another aspects of the process of realizing the conceptual structure in executable form.
Then I assert a mathematical proposition, which I think is very difficult to challenge. That is, if in 1986 any less than 9-10ths of the troubles, are accidents, then shrinking all the accidents to zero will not give you an order of magnitude improvement. Therefore, if there is a silver bullet—I picked the werewolf word because I say a lot of projects start out seemingly innocent and straightforward, and then in the dark of the moon they turn into monsters (and you may have had a project like that, I have). Therefore, if there is anything that offers a 10 fold improvement in productivity, it must address the inherent conceptual complexity and that may mean dealing with the concepts at a different level.
So there is the argument. Now the argument can be challenged on several grounds, one is – You could disagree with the postulate that you can divide things that neatly into the conceptual structure or something else. One is, you could say “Well, I believe more than 9/10th of the remaining troubles are accidents”, I have never had anybody assert that. But I asserted that opinion which is quite challenged. And one is, you could say the mathematics is wrong; I would just say that is foolish indeed; the mathematics is not wrong, and you know you have seen that in several forms.
Then I undertake to treat in the rest of the paper the various things that have been put forward as silver bullets for slaying the werewolf and try and explain why those seem to me to be dealing with accidents and not essence.
I also make a bold statement. That is: no one technique will appear in the next 10 yrs, that is ‘86 – ‘96, that gives 10-fold improvement in productivity or any of several other good things.
A burst of papers came and said “Oh yes, there is a silver bullet, and here it is, this is mine!” When ‘96 came and I was writing the new edition of The Mythical Man-Month and did a chapter on that, there was not yet a 10 fold improvement in productivity in software engineering. I would question whether there has been any single technique that has done so since. If so, I would guess it is object oriented programming, as a matter of fact.
The next panelist was Dave Parnas, who wrote the first works on designing systems with modules based on “hiding information” criteria and who has been a strong advocate for a software engineering to be treated (and accredited) as an engineering discipline (such as civil engineering, mechanical engineering, etc).
Dave quickly declared that there is no silver bullet for software and examined why this question is still hot 20 years later; Dave provided three reasons:
- People have a “very natural tendency to look for easy answers to hard questions, designing software is hard and it will always be hard.”
- New technologies are often over-hyped: “they have to kind of paint it as a silver bullet because otherwise people won’t listen.”
- Desire for people to seek better tools rather than “actually learning the trade.” Dave used the metaphor: “there is an old saying: ‘the poor workman blames his tools’ - people are poor workmen, I think most programmers are poor workmen.” Dave also said that people are looking for silver bullets to avoid learning the trade.
Dave challenged the notion that a lot of progress has been made in software engineering, instead attributing much of our advances to hardware. Dave Parnas concluded: “there are some things around which I call lead bullets, plain old ordinary bullets, disciplined, hard working, need a lot of training to use them. And we don’t do it and my question really is why don’t we use the lead bullets?”
Linda Northrop praised Fred’s article and mentioned some her main takeaways: “that software engineering involves more than programming…the hardest thing about building software is figuring out what to say, not how to say it.” Linda agreed that there is no silver bullet and that when her team did focus on essential complexity, “the results have been breathtaking.”
Linda cited Kristen Nygaard (co-inventor of object orientation) saying that Simula (the first OO language) was about modeling. In 2001 Nygaard said (in Linda’s interpretation) that we had “lost the essence, we weren’t thinking about modeling, we were all excited about languages and doodads and frills”. "In his mind," Linda added, "that didn’t have anything to do with the essence of object orientation."
Linda concluded:
The boundary between developers and those that we have, I think inappropriately call users, and our accidental innovations, help little. To wrestle future werewolves we still need great designers, and I think we still have far too few, and we still need to cultivate an atmosphere of hard work but also an inner-disciplinary perspective that takes us uncomfortably out of our coding world. Our world of what, if I could just borrow a phrase from this morning’s key note, “the technology push”. I think the reason we have the technology push is because we don’t do the hard work to understand the needs of who we are trying to address.
Aki Namioka from Cisco systems was next and suggested that we have seen great progress in the last few years because more and more people are now able to create useful applications (such as websites) without being programmers, “we have actually created tools that makes programming not just something that sophisticated engineers are producing but lots of people…”.
Dave Thomas, a key figure behind IBM Visual Age for Smalltalk and the Eclipse IDE, framed Brooks’ paper as a challenge for us to continually try to address productivity; however Dave called today’s state of the art in software development (especially middleware) a gratuitous disaster due to overly quick fast pace of change in API’s and frameworks which create immature software and also create too much difficulty for average software teams to track and stay up to date with. Dave claimed that the majority of today’s enterprise systems are basically just CRUD applications: “In the end these people are trying to do things that are fairly straight forward, and if they were working on a mainframe using a 4GL, they would actually have the thing done.“
Dave Thomas wanted to express his feelings about today’s universities being more concerned with certification than competence, “so we are certifying incompetence.” Dave lamented that new graduates are not taught a “broad spectrum of computational diversity… so now any time a new language comes out, these poor people churn and go on and on because they have no basic set of common concepts that allows you to understand these things.”
Dave Thomas suggested that lessons from engineering and manufacturing are the things that help large projects scale, more so than agile practices; however, Dave called “a breakthrough” that today’s developers think it is important to test.
Dave Thomas concluded that he has seen “real successes” in niches, citing highly specialized domain-specific and specialized language applications in airlines reservations and hedge funds; however, Dave cautioned that developer skills required are “very, very high” and cannot be “spread to the masses”.
Ricardo Lopez was up next and gave a spiritual dissertation positioning two constants in humanity’s history: fear and the need to optimize and improve. Ricardo positioned fear in this context as fear of the werewolf, who personifies software failure. Ricardo claimed that there is a silver bullet and it can come from two places:
- “Let’s go ahead an embrace complexity rather than run away from it because it’s actually, in my experience, the attempt to avoid complexity for fear of failure that actually leads to the increase in accidental complexity which guarantees failure.“
- “Whenever you seek to strive for the excellence that is within you to become both personal and professional, when you share that with your peers, you are the silver bullet. You are what it takes to get those magnitudes of improvement in anything you’re doing“.
Ricardo continued his empowering message:
When you have thought to improve yourself and pull up others around you by caring about them, caring about their growth both personal and professional have you not seen the increased productivity of your team? The whole art of collaboration is about personifying the shared experience of what you’re going through. This truly does give you an order of magnitude.
Ricardo concluded that “the silver bullet you’re looking for is inside.” Providing a personal example, Ricardo pointed out that with regards to the numbers of lives that were NOT lost during the California forest fires that were occurring at the time of the panel, compared to 100 years ago, was proof that: “We, as a civilization, have done several orders of magnitude improvement in what we are capable of, because we are silver bullets“.
Martin Fowler was the last in the row of panelists, and began by discussing how important and influential Brooks’ paper was on his life. Suddenly, Martin began to scream uncontrollably while grabbing his neck while falling out of sight behind the table. I threw my laptop to the side and leapt to my feet, intending to go and try to help when suddenly Martin re-appeared wearing a quite scary werewolf mask. The werewolf was the personification of failure in software projects, as Brooks’ described it at the beginning of his article. The following excerpt cannot really be appreciated excerpted, therefore, below is Martin’s full introductory speech:
I would like to make a few remarks about why I have survived so effectively. I agree with Mr. Brooks that object orientation is a very dangerous and evil idea, but I have managed to overcome it without too much difficulty. Certainly there are many good ideas in object orientation, but the great thing is, nobody actually does it.
At OOPSLA, yes, there is a lot of talk of objects, but you go out there into the rest of the world - nobody really does it! They use the languages - but they have no idea about the ideas. You might think you have succeeded with objects – but you have a long way to go to defeat me on that front.
And I have other weapons at my disposal! I love multicore concurrency systems. You are going to suffer so much with thread problems and race conditions, I'm gonna have so much fun over the next few years.
Those of course are part of the accidents, they are really not that dangerous to me. The real issues are those that attack the essence, and even so I have managed to survive.
One of the most dangerous things without a doubt is buying rather than building: the use of pre-built components and structures. It is a great theory, but it has one crucial weakness - it only helps you if the libraries are any good, and I have been very good at getting people to build very bad libraries.
Another key attack on the conceptual essence has been to grow great designers. I agree with my former friend here, that this is one of the most promising attacks and worryingly, many of you here understand this. But fortunately nobody outside of here really understands it.
People want software to be easy to do, they want software to be trivial and there is lots of people who are prepared to encourage this, and as a result they don’t take software development terribly seriously, and the very invisibility of software development helps even more, because people don’t understand how hard it really is, and that error of judgment means that people don’t value great designers enough.
The challenge is not to convince yourselves that you need great designers because you already know that. The challenge is to make that more visible to the broader population, and so far you have been completely pathetic at doing that.
Then the final point, with the two remaining parts of the Fred Brooks’ attacks on the conceptual essence, which I will sort of combine together, and it all has to do the way in which people go about building software: the process and the encouragement towards rapid prototyping and incremental development.
One of the great surprises for me over the last 30 years is how much I have come to adore waterfalls. It is remarkable, you humans, you seem to think that with hardly any information at the beginning of the project you can map out exactly what’s going to happen, pin down costs, commit yourselves to all sorts of unrealistic plans. And then you wonder why it is that I always manage to turn up. That illusion of control, the idea that on basis of hardly any data you can let make these grand predictions, that very folly is one of my greatest strengths. This is of course why it comes down that even lead bullets don’t hurt me.
One of the greatest things that helps me is the sheer desire for silver bullets. People have spent so much of time to trying to create silver bullets, so much of time trying to sell silver bullets, that they create far more work for themselves. The silver bullet purveyors, they are the great allies of mine. Thank you.
Panel impresario, Steven Fraser, then opened the floor to questions from the audience.
The first to speak from the audience was Bertrand Meyer who mentioned the fact that, according to his experience, many people, especially managers, rejected new ideas of the ‘80s and ‘90s like OOP, which actually were old ones, because they did not believe the new technology proposed to them would have a serious impact on development, and they used as argument the No Silver Bullet paper.
Fred responded by reaffirming that any technological improvement that deals with accidental difficulties won’t solve the real problem, even if it brings a 2-3 fold increase in productivity.
Linda said that she is not interested in marketers who want to sell their silver bullet, unless they offer a solution to the actual needs we have. Then she partially disagreed with what Ricardo had said earlier:
When he talked about the reasons for silver bullets, he seemed to have given all noble reasons for why we have silver bullets, that we have fear of complexity and that we want to increase productivity and this sheer daunting nature of complexity drives us to be afraid of it. I’d say that there is a ignoble reason why we trumpet silver bullets, and that’s greed.
John Roberts (Qualcomm) asked if a team made of greatest developers from all organizations would become a silver bullet or a recipe for disaster.
Ricardo supported the idea of having great people in a team without promising getting a silver bullet:
It is very important to get your best together, because they are not only the people taking you forward and tackling more essential complexity, and not falling prey to fearing it and therefore accidentally introducing the non essentially complexity, if you will. You are also providing opportunity for others to learn and that’s how we leverage each other, and that’s how civilization becomes what you and I know of it today.
On the same topic, Dave Thomas added:
So if you’re young and you want to find out how to be good, find the really good people that you can work for, and that includes executives that you can work with, because you learn from people who are better.
Dave Parnas talked about the subject of the previous question saying that someone who wants to sell you a couple of days training course is actually selling a silver bullet, and they should find buyers interested in one, inferring that he was not interested. A true, lead bullet is education, and that takes a lot of time of learning and training.
The Werewolf could not help himself but grinned:
Yes, it’s always very useful to me this idea that good people cannot collaborate together in teams. I can always sow some doubt by saying you can never get good people together, because there aren’t enough of them, which of course stops people from trying to get good people together. If you don’t try to get good people together, you’ll never succeed - so that’s an easy route for me to take.
Then I talk about the difficulty of getting good people to actually collaborate and that difficulty works well because people don’t try hard enough to help people to work together. They don’t understand that how much we humans need to get on with each other, how much work it is to actually have a team of people collaborating together. Again people want things to be easy and straight forward, and then, of course, I can always use the heavy weight process, that’s a very nice technique as well and this again ties into the question about new things.
Take the agile community for instance, I mean they are so cute, they run around talking about how to get rid of heavy weight process, then they forget there’s a whole group of people now who don’t really understand what agile is about. They focus on the syntax and not the semantics, and they’re completely undermining everything the agile community is doing, and that is very satisfactory to me.
Joe Yoder, the OOPSLA Panels Chair from The Refactory, added that good people, good design, collaboration, evolving requirements, all put together, might not be the desired silver bullet, but may help fight the werewolf off, and deliver the software in time.
Dave Thomas agreed with Joe about having the right mix of people and techniques, but added, based on his own experience, that we encounter a good mixture accidentally, that we don’t actually know how to do it repeatedly to be able to create it anytime, anywhere we want it.
Fred recommended reading the books Peopleware and The Carolina Way for those interested in leadership and growing great teams.
Linda talked about the difference between managing and leading. She also insisted on leading as coaches do it, building on character and leadership.
The Werewolf made the audience burst with laugh:
Peopleware is one of the most of the most dangerous books out there. Fortunately, it is hard work to manage a team and to focus on people's interactions. It is much easier to fiddle your thumbs and play with Microsoft project. Every time somebody creates a PERT chart or a Gant chart, I get to eat an extra kitten.
The panelists were then asked how the productivity could be measured to see if there was a 10 fold increase. The Werewolf was quick to sneer:
This is of course one of the greatest things I have at my disposal. You can’t measure your output, you can’t measure your productivity, no way you can run sensible experiments to figure out whether one technique is better than another. … Sometimes it’s just too easy for me.
Ricardo agreed with the Werewolf and covered some of the methods used to measure productivity, concluding that it is very difficult to do that.
Dave Parnas also sustained the same opinion on measuring productivity, and gave the following example: “If you give the same problem to 2 people, one of them in 2 days produces 2000 lines of code that does the job and the other in 3 days produces 500 lines of codes that does the job, which one do you want, especially if you have to maintain that code?”
Someone from audience asked when can we expect a proper development environment that allows the developers to focus on essence not on accidents?
The Werewolf was clear what he’s doing to stop that from happening:
The biggest difficulty in software development is the communication between the people writing the software and the people for who the software is being written. The conceptual attacks, the essence problems, focus a great deal on trying to improve that communication. The trouble of doing that, of course, is that it’s hard to get that communication flowing. Some of the business people don’t want to talk with software people, and fortunately the software people are hopeless when it comes to social interaction, they can’t talk to business people.
The key things that I do on projects is that I create as many barriers as possible between business people and developers: try to throttle the communications, have them talk to each other in documents, put them into different departments. Then what happens (I think one of the symptoms of this is very nice for me) is that the software people get so frustrated that they firstly lose interest in the business problem, and in which case I have won really. And they make it even worse because they sense they can’t focus on the business problem, so they start focusing on what they do know about - which is infrastructure stuff. And they build elaborate infrastructure because they can’t think of anything else to do.
Brian Foote (Industrial Logic) affirmed that the world is running on “bad code”, and asked why don’t we try being more effective at writing bad code and focus on making bad code better.
Ricardo, in his usual style, mentioned that humanity has never done something perfect and has learned to accept shortcomings, and it will be the same with software. We just need to get rid of the cataclysmic code, and try to avoid writing bad code, pretty good being acceptable.
Dave Parnas had a different approach to the issue:
What is good for the developers is probably not good for the world. I have known an awful lot of people who have made a reputation as a genius in their own company or department by writing something so complex that nobody else could fix it. I think we have to get away from that and I know it might lead to some unemployment of software industry and I would welcome it.
Another listener from the audience asked if there is something we can do to finish the OOP revolution.
Dave Parnas replied quickly that he will give a presentation later in the day that will address that specific question, but added “you won’t like the answer”.
Fred said that “the best way to make a system that is useful for a lot of applications is to make one that really useful for one, and then generalize it.”
Dave Thomas fired at software frameworks:
The best way to get software components is to stop people build software frameworks, where they let the raw material leak out with all the encapsulation violations and complexities of frameworks. One of the big mistakes we made, I believe, is encouraging people in building frameworks and shipping them. Frameworks are the best way to ship something that is unfinished and is not professionally polished.
Approaching the session end, Steven invited the panelists to conclude in reverse order, the Werewolf being the first to speak:
It is true that software development has made great progresses in the last 20 years, but to be honest, I am not harmed by the great progress that you have made, because my essence lies in my unexpected appearance. … Because human beings have this wonderful optimism when it comes to software, they think that it is always going to be so easy. And even if you got to be more productive, you still over estimate your capabilities and that’s where my strength comes, it is your inability to see the real world, that is my greatest strength.
Ricardo was optimistic about our ability to tackle complexity:
We have been embracing increasing complexity from decade to decade, from generation to generation, and will continue to do so, and the werewolves will get bigger and larger and uglier, but this one will pass.
Dave Thomas thanked Fred for the “grand challenge” offered, and concluded: “A goal is important to have, a vision is important to have, not achieving doesn’t really matter if you make the journey.”
Aki pointed out that our search for silver bullets has led us to create software systems that are used by people every day.
Linda expressed her appreciation for being on the same panel with Dr. Brooks and Dr. Parnas, and shared one important lesson she learned from life: “focus on the essence not the accidents”.
Dave Parnas took a stand and defended the waterfall software development process considering it has been unjustly criticized. He also advocated simpler systems implying that complex ones are not the best solution to our problems. He ended saying that we should not be looking for easy answers to hard questions.
Fred made the following observation: “I know of no field of engineering where people do less study of each other’s work, where they do less study of precedents, where they do less study of classical models. I think that that is a very dangerous thing”.
Steven Fraser closed the session by declaring: “There is no silver bullet!”
About OOPSLA
OOPSLA, the original conference devoted to object-oriented programming, has grown in scope to encompass programming, practices and paradigms addressing current software challenges, such as programmer productivity, security & reliability, ultra-large scale systems and evolving hardware platforms. At OOPSLA, you can find out about the latest developments from research and industry, participate in a workshop, improve your skills at a hands-on tutorial, or join other sessions that will feed your mind in creative and fun ways.