Software Development Has Always Been Difficult

In the previous article, we defined our own ten commandments for successful software development. In this article, we present some of the reasons why successful software development has always been so difficult in the past. The answer lies in the unique combination of people, processes, and technology that need to come together for a software development project to succeed. If you understand the dynamics of this combination, you will start to understand why there has never been, and never will be, any "silver bullets" in software development. This is a necessary starting point in understanding the difficulty surrounding successful software development. Only by learning from lessons of the past can we hope to avoid making the same mistakes in the future. Let’s take a brief look at the history of modern software development and identify some of the difficulties surrounding successful software development it.

In the 1970’s, development backlogs for corporate IT departments averaged eighteen months or longer. Since IT was usually the only department with the necessary resources to develop software, it owned the monopoly and wasn’t often concerned about service levels or prices. In the 1980’s developers struggled with PC’s, DOS, and 64K memory limitations. In the 1990’s, just as many software developers thought they were starting to understand client-server software, widespread use of the Web set expectations for point and click access to any piece of corporate data. Software and network infrastructures struggled to catch up with web technology that obsolete, literally overnight, many of even the newest client-server software architectures. One thing, however, has remained constant over time: there are no silver bullets in software development.

Successful software development starts with good requirements and good software architecture, long before the first line of code is ever written. Even then, since software is easy to modify, compared to hardware, it often is all too easy for users to change the requirements. The impact of changing a single line of code can wreak havoc on a program, especially a poorly designed one. On the people side, you’ll need more than just a few good software developers for a successful project. Besides good developers, you’ll also need system administrators and other support staff, such as database administrators in your development organization. As you schedule and budget a project, remember to make programmer skills the largest weighting factor, more so than the language, development tool, OS, and hardware choices you will also have to make. Finally, start planning for testing, production rollout, and maintenance of your software early in the project life cycle, or you’ll never catch up. If COBOL programmers in the 1970’s would have planned for users accessing their programs through a web-based front end in the year 2001, imagine where we would be today!

Software’s Difficult Past

In the 1970’s, IT departments running large mainframes controlled most of the corporate software development projects. The mainframe was the infrastructure for the enterprise-computing environment. COBOL was the language of choice and any department with adequate budget willing to wait for the average IT department-programming backlog of eighteen months could have the application they wanted developed or modified. Software was difficult to develop if for no other reason than because development was so tightly controlled by a small group of people with the necessary skills and access to expensive computers. In reality, much of the perceived unresponsiveness of centralized IT organizations was not due to any lack of software development skills or organizational structure, it was simply a result of the software architectures imposed by COBOL and mainframes.

Mainframe-based enterprise software applications, such as payroll processing, were typically large monolithic programs in which even simple changes were difficult to implement. The complicated structure of such programs usually limited the number of people who could modify them to their original developers. It was cost-prohibitive to have a new developer learn enough about a large mainframe program to modify it. This is painfully obvious today as many organizations return to the 70’s era code and try to update it to the year 2000 compliant. For this reason, development managers would instead simply wait for the original developers to finish their current task and then assign them to go back and modify their earlier work. COBOL technology was well understood by those developers who programmed in it. Even in the rather simplified model of centralized mainframe development organizations, however, people and process issues already played equal weight to technology issues in their impact on the success of software development.

In the 1980’s, inexpensive PC’s and the popularity of simpler computer programming languages such as BASIC led to the start of IT decentralization. Even small departments with no formal IT staff could purchase a PC, figure out the details of DOS configuration files, and get a department member with a technical background to learn BASIC. There was no longer always a requirement to wait eighteen months or more for a centralized IT organization to develop your software program. All of a sudden large companies had dozens or perhaps even hundreds of "unofficial" IT departments springing up, with no backlog to complete, which could immediately start developing stand-alone applications. The only infrastructure they needed was a PC and plenty of floppy disks for backing up their programs. Software seemed easy for a moment, at least until a program grew larger than 64K or needed more than a single floppy drive’s worth of storage. Even the year 2000 was only a far-off concern that crossed few developer’s minds. Most PC applications couldn’t access mainframe data, but most developers were too concerned about installing the latest OS upgrade to worry. Software development was still difficult; we were just too busy learning about PC’s to worry about it.

One result of the 1980’s PC boom on software development was the creation of "islands of automation." While the software program on a stand-alone PC might have been very useful to its user, such programs often led to duplicated work and lower productivity for the organization as a whole. One of the biggest productivity losses suffered by organizations was probably duplicate data entry because a stand-alone system could not communicate with a centralized system and the same data was required by both systems. Many organizations still suffer from the "multiple data entry" problem today and it continues to be a challenge to software developers who must reconcile different input errors when trying to collect and merge data. This process, referred to as "data cleansing", is especially true in one of the hottest new fields of software, data warehousing. Data cleansing is a well-known problem to anyone trying to build a large data warehouse from multiple sources. Electronically connecting "islands of automatic," rather than resolving the problem, simply increases the volumes of data that must be combined from various systems. As with many software development-related problems, the answer lies not in simply interconnecting diverse systems, but in doing so with common software architecture that prevents such problems in the first place.

In the 1990’s, corporations started to worry about centralized software development again. Microsoft Windows replaced DOS and brought a graphical user interface to stand-alone applications, along with a whole new level of programming complexity. Business managers realized that stand-alone PC applications might solve the need of one department, but did little to solve enterprise-wide business and information flow problems. At the same time, Unix finally matured to the point that it brought mainframe level reliability to client-server systems. This helped connect some of those PC islands of automation, but at a cost. MIS directors often found themselves supporting three separate development staff: for mainframes, Unix, and PC’s.

In the second half of the 1990’s, our kids suddenly started teaching us about the World Wide Web and the Internet. Almost overnight, network infrastructure went from connecting to the laser printer down the hall to down the hall to downloading multi-megabyte files from the web server halfway across the world. All it takes is a few clicks and anyone who can figure out how to use a mouse can get stock quotes and Java-enabled stock graphs on a web browser. A few more clicks to register on the site and you can be completing e-commerce transactions to sell or purchase that same stock. With the explosion of the Internet and its inherent ease of use, the same expectations for accessing corporate data, upwards of 80% of which is still stored on mainframes, were instantly set. Fewer computer users than ever before understand or even care about software development and its accompanying infrastructure. Software development, however, continues to be very difficult, and mostly for the same reasons.

Year 2000 and Other Similar Problems

Software has always been very difficult to develop and even more difficult to modify. Witness the billions of dollars being spent by corporations worldwide to upgrade or replace approximately 36 million applications so they will function correctly in the year 2000 (Y2K) and beyond. Those unfamiliar with software development struggle to understand why something as simple as the representation of the year, a four digit number comprehend by most kindergarten children, can wreak such havoc on software. Given software’s difficulty in handling the Y2K issue, it is even more amazing that brand new computer programming languages like Java can help accomplish such feats as bringing color images back from a small toy-like rover on Mars and allow them to be displayed on our PC’s at home a few minutes later.

Many people think the Y2K problem is a one-time occurrence in the history of software. This is not all so. Some other similar software problems include:

• Around the year 2015, the phone system is projected to run out of three-digit area codes, requiring changes to approximately 25 million applications


• In 1999, European countries switched over to a new universal currency, the euro, for non-cash transactions. By mid-2002, the use of the euro will be expanded to include cash transactions. These changes will impact approximately 10 million applications.


• Around the year 2075, United States social security numbers, based on a nine-digit number, are expected to run out. Approximately 15 million applications use social security numbers and would be impacted by this.

It is Hard to Structure Development Organizations for Success

There are certainly more wrong ways to structure a software development organization than there are correct ways. No single organizational structure, however, will meet every company’s needs. Centralized development organizations are often too big to be responsive to departmental concerns. Decentralized organizations may not have enough staff to provide needed specialty skills. Nevertheless, certain organizational concepts apply no matter how you structure your developers. For instance, integrated software development teams, where software architects, developers, testers, and other specialists are teamed together, almost always have fewer barriers to success than more traditional "silo" organizations. In the latter, software architects, developers, and testers are divided into separate teams and hand over the project from one step of its life cycle to another. There are several problems with this type of organization. First, it is not conducive to iterative development processes. Secondly, since no group has ownership in the other’s products, there is a natural tendency to blame problems on the work of another group. In our book titled Software Development (Chapter 6) focuses on organizing your software development organization for success and provides more information on these and other organizational topics.

It is Hard to Schedule and Budget Correctly

While entire books have been devoted to software development scheduling and budgeting, it remains rare to find a software development project completed under budget and on schedule. One reason is that development managers often set software schedules and budgets early in a project’s life cycle with little or no buy-in from the actual developers. Another reason is that many software development projects begin with pre-set budget or schedule limitations and then try to back-into the eventual end product. The best single piece of advice we have is to avoid using historical "magic numbers" from other projects when developing your budget or schedule. Accurate software development scheduling and budgeting requires that you understand the project, know the developers, development environments, and other factors that will impact your schedule and budget. These issues are addressed in Chapter 12 of our Software Development book.

It is Hard to Select the Right Language and Development Tools

Language choice continues to have a major impact on software development projects, starting with the software architecture. For a given project, the software architecture will look quite different if FORTRAN is chosen as the development language than if the Java was chosen. Chapter 14 discusses some of the features and benefits of today’s most widely used programming languages. Combined with other information in this book, the reader should be able to quickly narrow the language choice for any single project to one or two languages. Once a language is chosen, you will also need to select one or more development tools. Many development tools start by having you design the user interface and thus focus on that task. Mapping the user interface to the back-end database and adding business logic is left as an exercise to the developer. Other tools start with the database design, and use this to derive the user interface and structure the business logic. In both cases, the developer is forced to trade off one design for the other. A better approach, although supported by fewer development tools, is to start by defining your business logic. Once the business logic of an application is designed, it is then much easier to derive an appropriate user interface along with the required back-end database structure. Chapter 15 discusses the features to look for in a development tool and describes the various kinds of development tools available.

It is Hard to Select the Right OS and Hardware Platform

In the future, platform independent languages like Java might make OS and hardware issues irrelevant. Today, however, the OS and hardware platform chosen continues to have an impact on software development. Chapter 16 discusses general requirements for hardware environments, including developer desktops, servers, and production hardware. Also discussed in this chapter are hardware architecture issues, such as SMP Vs. MPP, and their impact on software architecture.

It is Hard to Accomplish a Production Rollout

One of the most overlooked reasons for the failure of software projects is the difficulty associated with a successful production rollout. From bringing online a new corporate-wide financial system to upgrading a simple software application, successful production rollout does not happen without careful planning and execution. Some of the best designed and developed software applications never see production use because they did not take into account some important factor of the production environment. Chapter 13 presents our solution to the production rollout problem, the Web-Centric Production Acceptance process. As with many software development issues, planning for production operations needs to begin early in the software life cycle as the software architecture is being defined. Many chapters in this book therefore will contain information related to helping you accomplish a successful production rollout of your software project.

http://www.harriskern.com/index.php?m=p&pid=377&authorid=34&aid=28