Interview and Book Review of The Technical and Social History of Software Engineering
Capers Jones wrote the book The Technical and Social History of Software Engineering in which he provides an overview of the evolution of information technology and software development.
The book starts by describing the human drive towards faster computation, followed by 9 chapters that cover developments in software engineering from 1930 until 2010.
Major software failures that have happened in these years are explored in a separate chapter. The final chapter of the book describes ongoing developments and provides a projection of the future until 2019.
InfoQ interviewed Capers about advancements and events in software engineering and the effects that they have had on our society.
InfoQ: What made you decide to write this book?
Capers: In 2006 after we had lived in Rhode Island for six years I wrote The History and Future of Narragansett Bay, which covered local history from before the last ice age through modern times. It was fun to write. Since I have worked in software since 1965 I thought it would also be fun to write a history of our industry, and it was enjoyable. Out of my 16 books 14 are technical books on software management topics and two are histories, Narragansett Bay and The Technical and Social History of Software Engineering. I may write another history book on “lost technologies” such as Incan stone work and the antikythera mechanism.
InfoQ: Why did you focus on the "technical" and the "social" aspects, why these?
Capers: The main incentive for my book was Paul Starr’s 1982 book entitled The Social Transformation of American Medicine. His book won a Pulitzer Prize in 1982. Until I read it I did not know that medicine was even more chaotic and unprofessional than software. Medical schools used to have two-year curricula and did not require college or even high school graduation. Students never entered hospitals. There were no licenses or board certification and no way to distinguish malpractice. Harmful substances such as arsenic and opium could be prescribed. We have similar problems in software. Software has had a rich social history and has been a pioneer in gender equality compared to older forms of engineering. Many important social organizations have been created such as the ACM, IEEE software and computing groups; SHARE, GUIDE, and COMMON, social networks such as Facebook and Twitter, and many more.
InfoQ: Going back in time, when was the first software program written and what did it look like?
Capers: This is not easy because sequential instructions have been used in mechanical devices such as the Jacquard loom for hundreds of years. Analog sequences have been in use for thousands of year. Probably Ada Lovelace, for whom the Ada language is named, wrote the first sequence of instructions that might resemble modern programs. It was written for the analytical engine of Charles Babbage circa 1842. Ada was the daughter of the poet Lord Byron. When I entered programming myself we still used punched cards or paper tape and wrote either in machine language or basic assembly language.
InfoQ: Where does the term "software engineering" come from? Why "engineering"?
Capers: The term software engineering was used at the well-known NATO software engineering conference in 1968, partly to be provocative and partly to add status to software. In 1958 a mathematician and statistician from Bell Labs named John Wilder Tukey used the word “software” in a paper. This was in the context of being a separate entity from “hardware.” This is the first known use of the word “software” in a computer context.
Also interesting is the word “computer.” This was a common term in the 1940’s but it was a job description for a mathematician who performed various kinds of calculations. Many of these human computers were women because in those days women were paid less. When the ENIAC computer was being finished in 1945 six women “computers” were hired to program it, and the term “computer” morphed from a job description for a human to the device that did the calculations.
InfoQ: Software engineering education has changed during the years, and is still changing. What is the current state, and what can we expect in the (near) future to happen?
Capers: When I entered college in 1956 there were no computers on campus; no computer science program; and no software engineering program. All of these evolved over the next decade and beyond. To learn programming I had to attend courses at IBM because none were offered at universities. Future education will probably be in the form of virtual reality campuses where avatars of students and professors meet and have access to every known publication and all leading tool suites.
InfoQ: Many people have lost their jobs when computer came in to automate the work. But IT also created new jobs. During the years, did we lose more jobs due to computers, or did the number of jobs actually increase?
Capers: The net loss of clerical jobs was smaller than anticipated because many clerks who handled paper documents became data entry personnel. Software has created more jobs than have been eliminated, including entirely new kinds of jobs such as agile coaches, data base experts, and software quality assurance specialists. Today we have about 128 occupation groups involved with software in large companies.
InfoQ: What is the biggest software system that has ever been built?
Capers: This question as stated is not answerable because countries such as Russia and China are not going to tell us about their largest systems. Several defense applications have topped 300,000 function points and several commercial ERP packages are in the 250,000 function point size range. My taxonomy runs like this: < 10 function points are very small; 10 to 100 are small; 100 to 1000 are medium; 1000 to 10,000 are large; 10,000 to 100,000 are very large; over 100,000 function points are extremely large.
InfoQ: What made it possible to develop and maintain such a large system and complex system?
Capers: Quality control and change control are weak links. Architecture and design are weak links. Gathering and analyzing requirements for massive applications is also a weak link. The industries that solved these problems first were those that build and operate complex physical devices such as computers, airplanes, and telephone switching systems. They all have top-notch quality control and top-notch change control. Banks, insurance, and companies that deal with information rather than physical devices have lagged in quality and change control.
InfoQ: There have been several major quality problems with software systems. Can you name some? What have we learned from examining those issues?
Capers: Quality problems are endemic. The most basic problem is depending too much on testing and not using effective methods of defect prevention and pre-test defect removal such as static analysis and formal inspections prior to testing. Also a common failing is not measuring quality. I use defect removal efficiency (DRE) or the percentage of bugs found before release. The current U.S. average is < 90% but best in class results are above 99%. You can’t get much above 90% DRE from testing alone. To hit 99% inspections and static analysis are needed before testing starts.
InfoQ: Your book mentions many software and computer companies. Can you name some that stood out, that were in some way different from the others? Did they survive being different?
Capers: The software industry has created new wealth far beyond anything the pioneers envisioned. Companies that combine wealth and technical innovations include Apple, Google, IBM, Microsoft, Oracle, and SAP although there are many others. We are far from being finished with either innovation or wealth creation as an industry.
InfoQ: Looking back, can you name some of the significant breakthroughs or innovations in software development? What have they brought us?
Capers: This is why I wrote a book that covers these breakthroughs in hundreds of pages. In a nutshell structured design and development, higher level languages, better methods such as agile, RUP, and TSP, and better quality control via pre-test inspections and static analysis. However manual programming is intrinsically expensive and error prone, so construction from standard parts to achieve higher levels of software reuse will become even more important.
InfoQ: Which technical or social topics do you expect to dominate software engineering in the coming 5-10 years? Why these?
Capers: The book itself has a chapter that runs from 2010 to 2019 with my predictions for future changes. Among these will be ever-increasing numbers of cyber attacks by both hostile governments and also by organized crime. On the good side will be new and interesting kinds of social networks; better training methods; big data; and increasingly powerful kinds of predictive analytics.
You can download an excerpt from the book here.
This interview is based on the book, 'The Technical and Social History of Software Engineering' authored by Capers Jones, published by Pearson/Addison-Wesley Professional, Dec. 2013, ISBN 9780321903426, Copyright © 2014 Pearson Education, Inc. For more info, please visit the publisher website.
About the Book Author
Capers Jones is currently vice president and chief technology officer of Namcook Analytics LLC. His background includes 12 years as a manager and research software engineer at IBM; four years as assistant director of programming at ITT; and 15 years as chairman of Software Productivity Research (SPR). He co-founded Namcook Analytics LLC in 2011 with his partner Ted Maroney. Capers has 16 published books and over 100 journal articles primarily covering software risk analysis, software quality control, software measurement, and software estimation. He has designed eight proprietary software estimation tools and four commercial software estimation tools.