Wednesday, December 27, 2006

Software Development's Evolution towards Product Design

Occasionally, some poor fellow at a dinner party makes the unfortunate mistake of asking what I do for a living. My initial (and quite subdued) response is that I help design software for artists.

Then comes the inevitable question, “Oh, so you are a programmer?” A gleam appears in my eye and I no longer feel obligated to blather on about the rainy weather. With a great flourish, I whip out my gold nibbed pen and draw a little diagram on a napkin that explains concisely how modern software development works. In the grand finale, I circle one of the little scribbles buried deep in the entire convoluted process and proudly proclaim ‘And that is what I do!”. This admittedly selfish exercise usually keeps everyone involved merrily entertained until dessert arrives.

After dozens of napkin defiling lectures, I’ve put together an extended PDF of my sketch for download. In short, we have a one page infographic that explains:

* The evolution of software development over four distinct eras.
* The key goals of software development and our saddest failures
* Where software development is moving in the future.

The diagram also contains a surprising amount of poo. But then, that is the bigger lesson lurking within the scrawls. Much of what software developers create fails to serve the full spectrum of their customer’s needs. The funny part is that the usually non-technical folks that I’m talking to laugh heartily at this point…they know exactly what I’m talking about.

The Technocrat Era: Programmers serving programmers

At the dawn of software history, programmers wrote software for other programmers. This was a golden era. Life was so simple. The programmers understood their own technical needs quite intimately and were able to produce software that served those needs. The act of software development was a closed circuit. A programmer could sit in a corner and write code that he wanted. By default it also happened to apply to other programmers.



Programmers that grew up in these idyllic days still remember it fondly. There is still the programmer who will claim that all they need is EMACS and the latest version of GCC to make great software. For software intended for a highly technical audience, they may well be right.

The Early Business Era: Programmers attempt to serve others

The end of the Technocrat era was came about due to a startling discovery: the vast majority of the world was composed of non-technical people. Artists, accountants, authors, history majors and other unexpected consumers roamed the prehistoric landscape outside the hallowed engineering halls.

A new class of software entrepreneur called a ‘business man’ came into existence. This new creature realized that the hordes of non-technical people had their own needs that might be served by a well written program running on one of these new fangled “personal computers.” The business man gathered programmers and told them to build software to solve things like balancing budgets or writing letters.

The software products that the programmers created were technical marvels. They provided practical benefits far beyond what was available. VisiCalc revolutionized finance. WordStar and WordPerfect forever changed the act of writing. Even the Quantel Paintbox changed the world of art.

And yet the recently converted users, who couldn’t live without the efficient, powerful and technically amazing new software, were curiously ambivalent. They liked what these new products did from a practical standpoint, but found them to be confusing and often quite irritating.

You see, the programmers treated their customers just like programmers. They made them memorize crazy expert keyboard conventions and loaded the product with dozens of obscure features. This is what the programmers wanted out of a piece of software, so they assumed that the customers must want the same.

Unfortunately, the customer needs were a different beast. Their needs could be roughly divided into two categories:

* Practical needs: They wanted a product that worked. For example, the product obviously needed to save time and money.
* Emotional needs: They also wanted products the possessed less tangible benefits. They wanted applications that treated them with kindness and understanding if they made a mistake. They even wanted to use products that were attractive and conferred status. The customers desired programs that appealed to the softer aspects of their humanity.

When emotional needs were raised, the programmers quickly determined that the customers were indeed “nuts.” The pure technocrats simply did not possess the broad skill set necessary to comprehend, never mind serve, the customer’s unexpectedly important emotional needs.

The Late Business Era: Programmers and artists meet and do battle

The software market had become quite competitive at this point. Business folks, experienced in the Jedi wisdom of more mature markets, reasoned that perhaps serving emotional needs could give their companies an edge. A few companies experimentally hired non-programmers such as artists, marketing and usability dabblers. I use the term artist quite liberally here since it captures that charming, hand-waving vagueness of all classes of “people people”



Oh, the suffering that resulted. The inevitable culture clash was a bit like unleashing wild dogs upon one another and then watching them sulk afterwards.

None of the freshly introduced team members spoke one another’s language. The artists talked about fluff like color and mood. The marketing people made outrageous requests with zero comprehension about technical feasibility. The programmers were suddenly enslaved by bizarre, conflicting feature demands that they did not understand. “Make it friendlier” translates poorly into C++ code.

Let’s take something as simple as making an interface more appealing.

* The artist would whip up a picture sporting rounded corners and more pleasing colors. They’d send it to over to the programmer with a hand scribbled note “Make this.”
* The programmer then would either A) State that an infinite number of programmers could never finish such a technical abomination or B) Recreate the image using rectangles and the preexisting color scheme.
* Everyone would then rage at one another about their general incompetence.

In these battles for dominance, the winners lost horribly. If the programmers got the upper hand, they produced software that, despite great technical accomplishments, was ugly, difficult to use and no better than currently existing products. If the artists got the upper hand, you ended up with lovely products that didn’t do anything worthwhile.

The best products came from those odd teams that managed to compromise. The technology was clumsy and the emotional benefits of the software shaky. But it was better than the crap that customer had to put up with before. The original Mac OS was a great example of this. Later versions of Windows also managed to address a few emotional needs.

The Product Design Era: Can programmers and artists learn to work together?

The clock of progress has moved forward once again. The competition ratcheted up one more notch, and it appeared evident that companies that fail to master the lessons of the last era would be punished by the customers. Our battle scarred software companies were left looking for a better way.

Obviously, designing for emotional needs in addition to technical needs was a winning evolutionary strategy. In the last era however, mixing multiple potent skill sets together left organizational and cultural wounds that often were difficult to heal.

The companies that survived the influx of designers and marketing folks often relegated the survivors to their own separate silos. Marketing people got one org tree, developers got another. Artists and usability folks in most cases were shoved in random back corners to rot in black despair. Many groups were so busy protecting their domain that it was surprising that software got released at all. Release dates slipped by multiple years as the development talent stagnated is a cesspool of misplaced process.

What was needed was the most dramatic of transformations: A change in the cored development process. What made it difficult is that original culture of technocratic software development would need to evolve to support a broadly humanistic approach to product design.

The lead users of the Product Design Era

This change found root, as is the case for most dramatic transitions, from the most unlikely places.

There is a concept in product design known as the ‘lead user’[1]. This is a group of users that solves a difficult program far ahead of the mass market. Often they’ll cobble together their own tools and discover fundamental process and technology issues years in advance of mainstream users.

The old joke about writing Shakespeare with infinite monkeys randomly typing on typewriters got it all wrong. Instead give me a million customers trying to do their job with broken tools and one of them will stumble upon a process that is truly better. By watching the edges of the market place, we gain great insight into the direction that the larger market will take in the future.

The lead users in software development came from several widely divergent areas.

* The first was the game industry. Here, small cross functional teams built products focused entirely on serving emotional needs. Due to intense competition and vicious delivery cycles, many teams were forced to innovate far outside the traditional software development methodologies.
* The second were companies like Apple that followed curious ‘design’ philosophies more similar to that pursued by consumer good companies than software companies. What do shampoo companies and software development have in common? A remarkable amount it turns out.
* The third were web design companies. Due to the low cost of entry and the early emphasis on the web as a marketing medium, the web design market is dominated by tiny, hungry graphic design firms. They brought with them a culture of small teams, close collaboration between artists and programmers, and a nearly slavish devotion to serving their customers.

What the next era looks like

Each of these lead users follows a variant of what is broadly known in more mature industries as “product design.” They see software development not as a pure technical exercise. Instead they look at it as an integrative process of building a new product that solves both their customer’s combined emotional and practical needs. Even when forced to use a “technically inferior” platform, the religious devotion to rapidly and effectively serving customer’s complete spectrum of needs make their product offerings more attractive than the competition.

Robert G. Cooper, a well known researcher on new product development, states that there are several core factors [2] (listed in order of importance) for any successful new product design process:

1. A unique, superior and differentiated product with good value-for-money for the customer.
2. A strong market orientation – voice of the customer is built in
3. Sharp, early, fact-based product definition before product development begins
4. Solid up-front homework – doing front end activities like market analysis well
5. True cross functional teams: empowered, resourced, accountable, dedicated leader
6. Leverage – Where the project builds on business’s technology and marketing competencies
7. Market attractiveness – size, growth, margins
8. Quality of the launch effort: well planned, properly resourced
9. Technological competencies and quality of execution of technology activities.

Many companies in the Late Business era already emphasize a few of these factors. However, there are some differences. Notice how technical competencies are important, but last on the list. Notice also, how that creating a solution to customer needs is first on the list.

Software companies that understand product design tend to pour their efforts into the following activities:

* Focus on a unifying team goal built around customer needs. They ensure that the product is always driven by customer needs, not internal whims.
* Work together in cross functional teams. They build an organization that encourages the sharing of skills to promote problem solving. They discourage the formation of silos of individual ‘experts’.
* Communicate clearly with a process that applies to all team members. They build a commonly followed process includes both fuzzy front end activities like design and production activities like coding. This process forms the language that unifies disparate groups.
* Work efficiently with design friendly tools. The team avoids custom coding and ‘tossing it over the fence’ by adopting tools that work on common data format across all skill sets.

Benefits of a product design philosophy

The benefits of a product design process are well documented. New products that deliver superior, unique benefits to the customer have a commercial success rate of 98% compared to 18.4% for undifferentiated products. These products reach an outstanding 53.5% market share.

Some of the highlights of a strong product design include:

* Create highly competitive products that achieve market dominance.
* Save money by focusing on the right features that bring customer value, not low yield ‘nice-to-have’ features.
* Create passionate customers that accelerate the spread of your marketing message.

To the losers, the success of their rivals appears miraculous. “How is it that a slow web app can take away market share from our superior desktop application?” they ask in surprise.

The answer will be simple. The successful company identified the correct emotional and practical needs of the customer and poured their efforts into serving those needs. The richer companies - flush with silos of ‘wise’ experts - fought with one another and threw random features at the customer. It is rarely about doing more; it is about grokking customers and doing the small set of correct things necessary to succeed.

Dangers of the Product Design Era

As with any process transition, some groups are adopting these techniques slower than others.

* Even within game development there are huge swaths of publishers and development teams that are ignorant of techniques used to incorporate market research, concept testing and new-to-the world innovation into their process.
* Many web developers still create new products by following their ‘gut’ without clearly identifying their ultimate customer.
* Most traditional desktop software developers are just barely escaping the Late Business era of functional silos and warring factions.

Most laggards who hold desperately onto their aging processes will die off in the face of advancing competition. Those with larger war chests merely buy a small period of additional learning time.

Unfortunately, many companies that attempt to adopt a product design philosophy will also fail, despite their best efforts. Cultural change is hard work. To adopt product design you must alter the most basic DNA of the company’s values.

* It involves asking vice presidents to give up their empires that they’ve fought decades to establish. What is the point of an organization of engineers when engineers are all just members of small cross-functional teams?
* It involves asking the men and women in the trenches to give up their own dreams of building their own empires according to the old rules.
* If the ultimate reward of the old system is an isolated corner office with windows, try convincing people to work together in a common war room that has walls covered with whiteboards.

Such traumatic change is absolutely necessary. If you want to make great products for happy customers, you need to make the transition to a broader product design methodology. We may have once been a clique of technocrats. But now we must take our place in the broader society by providing human solutions to the very human customers that we serve.