Why Software Engineering Isn’t Like Other Engineering Disciplines and How it Changes the Game

It has been estimated that there are over 11 million professional software developers world-wide as of 2014. When I started as a programmer in 1973 one of the greybeards in the first company I worked for gave me some advice. He said, “Learn the things that never change.”

When I started college six years earlier in 1967 the school I attended didn’t have a major called Computer Science and so I did my undergraduate and graduate work in Mathematics taking a few computer programming courses along the way. This was the way many of us got started as software developers back in the 70’s.

The term Software Engineering was new at the time, being coined at the 1968 NATO Software Engineering Conference. The thinking back then was that we needed to apply existing engineering methods to software development to address common budget, schedule and quality problems that were being referred to at the time as the “software crisis.” As a result, what most people have come to think of as Software Engineering involves activities which greatly resemble other engineering disciplines including civil, mechanical, and electrical engineering.

On the surface this idea seems to make sense. When you build something using the other engineering disciplines (e.g. a bridge, a building, a specialized piece of hardware, an electrical circuit board) you need to figure out the requirements, design a solution, implement it, and test it. All of these steps make sense for software as well. So one could certainly argue from this perspective that software engineering should resemble these other engineering disciplines. However, when you look more closely at what we have learned about software development over the last forty years, as well as how we teach it to today’s software developers, this analogy quickly breaks down.

By the time the 1990’s rolled around, because computer programming had become such a big part of what was called Computer Science, many Universities had added a course with a title of “Software Engineering” to their Computer Science curriculum. Popular textbooks that were used at that time to teach these courses included Ian Sommerville’s textbook titled: “Software Engineering”. From 1992 to 1994 I used the Fourth Edition of this textbook to teach Software Engineering at Binghamton University. Today, Ian Sommerville’s textbook is still in use in many Universities around the world-now in its Ninth Edition. This leads to a question:

Why do we need to revise a textbook approximately every 3-4 years that supposedly is teaching our students the fundamentals of Software Engineering?

If you look at textbooks used in Civil Engineering, Mechanical Engineering, and Electrical Engineering the vast majority of these books do not require revisions nearly so often. To understand why this is the case we need to look more closely at what is being taught in most Universities around the world under the name of “Software Engineering.”

When you do look more closely you will find that we are teaching our next generation of software professionals whatever is currently popular in terms of software practices, and methods. Popular software practices and methods today are known by buzzwords such as Agile, Use Cases, User Stories, RUP, XP, Scrum Lean, PSP, TSP and the list goes on and on…

The problem with this approach to teaching Software Engineering is that software practices and methods frequently come and go and will continue to come and go which is why Sommerville must continually update his textbook. This leads to another question:

What about that greybeard in the first company I worked for in 1973 who told me to learn the things that never change? Did he give me bad advice? If not, what are we teaching our next generation of software professionals with respect to the things that never change about Software Engineering?

Before answering these questions, let’s first step back and ask a few different questions:

Does a set of things that never change in Software Engineering actually exist?

If they do exist, do we know what they are?

If we do know what they are, are we teaching them in a consistent way to our next generation of software professionals so when they come out of the University they are prepared to conduct themselves as software professionals?

Such a set of software engineering essentials does in fact exist. This belief has motivated an international group of volunteers to take on the task of codifying those essentials. The intent is for these essentials to be taught to our next generation of software developers helping to prepare them as true software professionals.

The volunteers involved in this initiative (known as SEMAT – Software Engineering Method and Theory) have been working on this task since 2010. This past year SEMAT achieved a major milestone with the announcement by the Object Management Group, an international standards consortium, that they have adopted “Essence” as an official OMG standard.

So this leads to a few more questions:

Just how different is the Essence standard from what is being taught to our software developers today, and has been taught for the past 40 years under the name of Software Engineering?

And:

Will the differences really help with the problems that many believe still plague the software industry with respect to common budget, and schedule over-runs and poor software quality?

From one perspective what Essence captures is not new. The Essence standard includes common words such as, Stakeholders, Opportunity, Requirements, Software System, Team, Work, and Way of Working. But from another perspective what Essence captures is dramatically new. In fact, some are calling it a “paradigm shift” that many of the “old guard” will have great difficulty even comprehending.

To give you an idea of the changes involved when using Essence I again think back to my early days as a programmer in the late 1970’s. In those days I worked in the flight simulation domain developing software systems to train pilots to fly high performance aircrafts. One of my areas of expertise was writing software to provide record/playback capabilities to help instructors train young aircraft pilots in flying skills.

I recall one specific project I worked on and a customer pilot instructor I worked with. After explaining to him how he could use my record/playback software to help him demonstrate to his student pilots where they had made mistakes, he excitedly wrote up a number of defects requesting changes to my software.

I argued vehemently with my program manager that none of these issues were actually defects. Because I had taken the time to explain what was possible with my record/playback software the pilot instructor began to envision additional features that could make his job easier. He wrote his ideas up on a defect form even though they were all enhanced capabilities we never planned to deliver and were not part of the requirements.

But my project manager didn’t want to discuss with the customer whether or not these requests were in-scope, or out-of-scope. His view was– as many viewed software then and still view it today– that it is easier to change software than engaging the customer in a discussion.

Because software is soft, we tend to view it as easy to change. It’s not like hardware. Metal isn’t easily bent. This perspective changes the whole game when it comes to software.

This ability to change software code quickly and in endless ways completely changes the dynamics that exist between software developers and their stakeholders including program managers and customers. One way this difference exemplifies itself is as users become familiar with the software they often see new ways that changes to the software could make their job easier as my pilot instructor customer did back in the late 1970s.

We now know from experiences that there are other dimensions to Software Engineering that are critical to effective professional software engineering practices. These other dimensions take us beyond just the ease with which the code can be changed. To date, these additional dimensions have not received anywhere near the attention they deserve.

When you change code you may also be affecting the requirements, and you may also be affecting other capabilities in the software system previously tested. Changing code means additional work, additional testing, possibly changes to supporting user manuals and so on… All this affects budget and schedule, and introduces additional risk to the quality of the software.

While on the one hand the ability to change the software code rapidly brings great power to the software industry, it also means that software professionals must be increasingly attune to their agreed way of working, the impact and time that it takes to do the additional work, and the risk when making unplanned rapid changes. The agile movement over the last ten years has provided a great service to help the software community understand this major difference related to Software Engineering including the importance of early and ongoing interaction with stakeholders and the importance of software developers estimating the cost of their own work.

While the software engineering community has learned a great deal from the other engineering disciplines, they have also learned the critical importance of these other dimensions that bring differences from previous engineering experiences. These differences mean that software developers need to be trained in new and different ways to be effective software professionals.

Shortly after the kickoff of the SEMAT initiative in March of 2010, one of SEMAT’s initial signatories sent me a draft copy of a book he was working on to review. Watts Humphrey who had planned to be very active in the early SEMAT work fell ill just as the SEMAT work was gearing up and I was asked to help him get his planned effort going. In late August that same year Watts sent me the following email just a few months before his passing. He agreed that I could share this email with others:

Paul, From your comments, it sounds as if you did get the point of my book, for which I am grateful….the correct answer and the one that I was most interested in pursuing with SEMAT, concerns how we can ensure that software professionals are properly trained and have a suitable set of professional attitudes and skills before they even get to industry. It is my hope that the SEMAT effort eventually will be able to spearhead the drive to get the academic community to refocus their programs on teaching software professionals to act like professionals and to manage themselves.

When they do, their graduates will be able to negotiate with their management and to do superior work…. That is what professionals should do… A good start in this direction would be to convince them of the necessity of having software people measure their own work. Since software work is, as we said, knowledge work, any truly accurate measures must be taken by the software professionals themselves. …Watts Humphrey

Watts Humphrey has been referred to as the father of software quality. After completing a distinguished career at IBM he went on to become a fellow of the Software Engineering Institute founding the Software Process Program. In 2003 he was awarded the National Medal of Technology.

Today Watts would have been heartened by the SEMAT work that is going on in the academic community. The first full University course based on the new Essence standard has been developed and is being delivered to students this year by Dr. Carlos Zapata at the Universidad Nacional de Columbia in Medellin, Columbia, and Essence is being used in first- and second-year software engineering courses at KTH Royal Institute of Technology in Sweden under the guidance of Dr. Mira Kajko-Mattson. There have also been Essence field studies conducted with students by Dr. Cecile Peraire at Carnegie-Mellon West in the United States. The next step for the SEMAT community is to demonstrate how Essence can help in industry by publishing case studies of actual use and measured results on industrial projects.