Wednesday, November 29, 2006

Success/Failure Criteria: Some Surprises

At a breakfast seminar here June 6 on "Factors for IT Project Success and Failure," Prof. June Verner of NICTA (the National Information and Communication Technology institute of Australia) provided a fascinating mix of surprises and predictables related to her subject topic. The findings came from NICTA’s study of 400 projects in the U.S., Australia, and Chile, using questionnaires and interviews to discuss success and failure factors with practitioners.

What were the surprises?

* Most projects that had no schedule were successful
* Requirements are needed for project success, but not necessarily early in the project
* Projects often continue successfully for some time with unclear requirements
* The choice of requirements methodology does not matter; UML was "no help"
* Using a development methodology was a success factor, especially when it was "appropriate to the application"
* Very few projects use risk management, and those that do rarely manage those risks
* Post mortem reviews are rarely held, and when they are it is almost always on successful projects
* In the U.S. (but not elsewhere), developers are involved in project estimation only when there are poor requirements (Verner speculated that this is because the powers that be were looking for someone to blame!)

And the predictables?

* Success comes from a culture that investigates and deals with problems
* The vision for the project (its business goals) must be shared among all project personnel, especially the project manager
* Project managers should be involved in the estimation activity
* Project managers should be good at customer and developer communication; they need not be good at the technology

There was some interesting data from the study, as well:

* 60% of organizations have no process to measure benefits
* 86% of projects had a business case, but 60% ignored it
* 33% of projects said they had no risks, but 62% of those failed
* 49% or organizations have had (one or more) project failures
* In one-third of the projects, the project manager had no say in schedule/budget targets
* 75% of projects were underestimated, none were overestimated
* 5% of projects had no project manager; 16% changed project manager at least once (and that was correlated with project failure)

Verner also asked developers what their criteria were for project success. They said:


* They had a sense they had delivered a quality product
* They had a sense of achievement
* They had enough independence to work creatively
* The requirements were met and the system worked as intended

Who Owns Your Software? Software Copyright and the Work for Hire Doctrine

Businesses all over the United States hire software developers to create software that offers a competitive advantage or cuts operating costs. Frequently both business owners and software developers enter into these agreements to develop software without addressing the issue of copyright. How does copyright law apply to these kinds of agreements, especially in cases where copyright ownership is not addressed explicitly? Who owns the software?

Think about a car repair shop owner who hires a developer to create a program that allows the shop to keep track of when its customers are due for scheduled maintenance and automatically generates a reminder email to that customer that it is time for the maintenance. The owner and the developer negotiate a $20,000 price for producing the program, and the developer agrees to have the project ready for installation in three months time. The parties send correspondence back and forth agreeing to these basic terms but never address copyright ownership.

The developer creates the program at his home, installs it on time, and receives his $20,000 fee. The program exceeds all expectations, customers rave about the customer service, and profits rise. The repair shop owner brags about this new service to his competitors. Several of them inquire if they might purchase the program and thoughts of a yacht and early retirement flood through the owner's head.

The shop owner is about to sign the first agreement to sell the software when he receives a letter from the software designer stating that the owner has no right to sell the software since the developer, not the shop owner, owns the copyright. The shop owner never thought about the copyright but thinks he must own it since he paid $20,000 for the program. Only at this point do the parties contact their respective lawyers for advice on who owns the copyright.

Copyright ownership is critical, since the copyright owner will have the exclusive right to reproduce, distribute, and create a derivative work (among other rights). Under our facts, if the shop owner does not own the copyright, he does not have the right to reproduce the software to sell to others. If he did so without the software developer's permission, the developer could sue him for damages arising from copyright infringement.

The Federal law addressing this situation is entitled The Copyright Act of 1976, 17 USC 201(a). The general rule is that the author of the work owns the copyright. The Copyright Act, however, contains an important exception called the "work for hire" doctrine. If the facts establish that the "work for hire" doctrine applies, the person for whom the work was created (in this case the shop owner) would own the copyright. The "work for hire" doctrine applies when employees create works within the scope of their employment or a situation where a certain type of work is specially ordered or commissioned by which an express agreement is to be considered a work for hire [Freiburn, 2004].

In our case, the bare facts indicate that the software designer is not the shop owner's employee. As a result, the employee portion of the "work for hire" doctrine does not apply. What about the second portion of the "work for hire" doctrine? Might it apply and revive the shop owner's dreams of early retirement?

Applying the second portion of the "work for hire" doctrine requires that the facts establish the following three conditions:

1. The work must be specially ordered or commissioned. This simply means that the shop owner hired and paid the designer to create something new, as opposed to paying him for an existing work [Jassin, 2004]. That appears to be the case here: the shop owner hired the software designer to create a new software program.
2. Both parties agreed, prior to beginning the work, that the work would be considered a work for hire. In our case, the parties did not address this point at any time prior to the software designer beginning the work. [Jassin, 2004]
3. The work must fall into at least one of the following nine categories under the Copyright Act:

1. translation;
2. motion picture contribution or other audiovisual work;
3. collective work contribution (magazine article);
4. an atlas;
5. a compilation;
6. instructional text;
7. a test;
8. answer material for a test; and
9. supplemental work such as a preface to a book.

In our case, software does not fall under any of these categories. In other words, even if the parties had agreed to designate the software as a work for hire prior to the work beginning, it would likely be invalid because software cannot be designated as work for hire since it does not fall within any of the categories listed above.

The software developer is the copyright owner. Unless the shop owner gets the developer’s permission, his yacht will remain firmly anchored in the showroom, and he will be at work bright and early Monday morning.

The issue raised in my fictional story could have been avoided if the parties had sought legal advice prior to negotiating the agreement. The copyright ownership issue could have been addressed with the assistance of legal counsel in preparing the agreement.

From the shop owner's perspective, the agreement might have read that the software developer assigned his copyright ownership to the shop owner as part of the agreement. Although the developer might be reluctant to do this or might require an additional fee to give up this right, the issue is on the table up front for resolution. In most legal disputes, use the rule of thumb that employing counsel to resolve the problem raised in this article will likely cost at least three times as much as if the parties had used lawyers to raise and resolve these issues initially.

Copyright law is an area that regularly defies the mathematical logic used to create the software programs to which the law applies. When your livelihood or business revenue depends on your intellectual property ownership rights, an ounce of legal prevention averts the expenditure of three times the amount for prescribing a legal cure.

Improving Developer Productivity With Domain-Specific Modeling Languages

According to Software Productivity Research, the average productivity in Java is only 20% better than in BASIC. C++ fares no better than Java [SPR, 2005]. In fact, with the exception of Smalltalk, not a single programming language in general use today can show a substantial increase in productivity over BASIC.

So after all of the language wars, objects, components, and frameworks, we are still scarcely more effective than 20 years ago. However, go back a couple of decades more and there is a radical change: a leap in productivity of 400% from Assembler to BASIC. Why was that leap so big, and how can we achieve it again today?

The 400% increase was because of a step up to the next level of abstraction. Each statement in C++, BASIC or Java corresponds to several statements in Assembler. Most importantly, these languages can be automatically translated into Assembler. In terms of productivity, this means you effectively get five lines of code for the price of one.

Did UML Increase Productivity?

Traditional modeling languages like UML have not increased productivity, since the core models are on the same level of abstraction as the programming languages supported: When designing in UML we still work with objects, their attributes and return values. One day of modeling with a UML modeling tool produces a day’s worth of code.

However, after using the code generation functionality of these types of tools you still need to go in by hand and add or edit the majority of the code. You’re thus trying to maintain the same information in two places, code and models, and that’s always a recipe for trouble. Ever seen anybody hand edit Assembler and try to keep their C++ code in synch with it?

Of course, UML has its benefits: the visual presentation can be read much faster to get an overview. But one look at the ongoing development of UML speaks volumes. Half of the developer world finds that UML cannot represent what they need in their models, and so want to add something to it. The other half says UML is too complex, and want to reduce it down to its core elements. UML tries to be all things to all men, and thus cannot raise the level of abstraction above the lowest common denominator.

Raising Abstraction With Domain-Specific Modeling (DSM)

How then can we raise the level of abstraction beyond today’s 3rd generation programming languages? One way is to move away from trying to specify all kinds of applications using only one generic set of program language concepts. Instead, each company could use the concepts of the products it makes, giving each its own visual representation.

Symbols in such a domain-specific modeling language (DSM language), along with the rules about how they can be connected and used, would thus come directly from the problem domain—the world in which the application is to run. This is a whole level of abstraction higher than UML. This results in a very expressive, yet bounded, design language that can only specify applications in the problem domain that it was designed to cover.

For instance, a DSM language for developing mobile phone applications could make use of concepts like "soft button", "menu", "send SMS" and "notification", which are descriptive of the mobile phone domain but can hardly be used for developing web applications, ERP software, or a business intelligence application. This narrow focus makes defining a DSM language a lot easier than a more generic language like UML or Java. The improved expressiveness on the other hand makes it possible to define an application completely with a relatively small modeling effort.

Because of this completeness, a good DSM language is well suited to be the basis for automatic generation of full code. This leads to a situation where you can have graphical models that form the core of your software development effort, and automatically generate all necessary code and documentation from them. The models are then both the design documentation—a view of the product or system at a high abstraction level—as well as its implementation in code. Documentation and implementation remain synchronized throughout the whole lifecycle of the application, and software development becomes truly model-driven.

Full code generation from a DSM language requires a code generator. The mapping from model to code is defined in the generator. DSM languages and the way we write code differ from domain to domain. A DSM language therefore requires a matching code generator that also meets the requirements of the problem domain. In other words, to make sure that you are able to generate full code—written in the way you want it—from your DSM language, you need complete freedom in defining how your language maps to code.

Vendor-proprietary or fixed code generators are quite useless for a domain-specific modeling language. This becomes even more valid with the knowledge that both problem and solution domains evolve over time. Therefore, you need to be able to evolve your modeling language and code generator and don’t want to be at the mercy of a tool vendor doing this for you.

When and How to Implement DSM?

Domain-Specific Modeling requires a DSM language and a matching code generator. Although defining these is much easier than defining a generic modeling language, it is still not a task for every developer. It requires a good knowledge of the problem domain and the code that is written for it. This domain expertise is usually found in situations where we deal with product family development, continuous development of the same system or configuration of a system to customer-specific requirements (phone routing systems, CRM, Workflow, payment systems etc.).

Experienced developers who work in these types of domains know what the concepts of their domain are, are familiar with the rules that constrain them and know how code or configuration files should be written best. Sometimes it is one expert who possesses this knowledge, sometimes it is shared among a small group of people with different skills. These experts are therefore well qualified to define the automation mechanism that will make the rest of the developers much more productive.

Situations where we deal with a repetitive development effort are thus well suited for DSM adoption. A facet of repetitive development is the use of, or move toward a common product platform upon which product variants are developed. The product platform based approach to software development has its roots in the drive to better manage the complexity of offering greater product variety.

Key in this approach is the sharing and reuse of components, modules and other assets across a product family in order to develop new variants faster by assembling them from standard components. These standard components in a standard environment form the product platform, which raises the level of abstraction and at the same time limits the design space.

In most situations however, development teams use generic programming languages to develop software on top of this product platform. Why use a programming language that has been designed to develop any type of application when we are already working inside the boundaries set by the product platform? It is this product platform that is supposed to make things easier, so why not make optimal use of this by using a DSM language on top of the product platform? The code generator then maps the models to code that interfaces with the product platform.

But what if the product platform itself is subject to a lot of change, or several platforms or platform variants must be targeted? In these cases it makes sense to create a framework layer between the generator and the product platform(s). A change in the platform then only requires the expert to make a change in the framework layer. The code generator and DSM language that all other developers use can remain unchanged—and of course there is no need to change the models the developers have made.

This is in stark contrast to cases where applications are coded directly on a platform, when platform changes generally mean wholesale changes throughout most of the written code. Building a framework between the generator and platform can be a good idea even for a single platform. It can make creating the code generator a lot easier as many of the variation issues can be handled inside this framework layer, instead of having the code generator deal with all the details.

To summarize: DSM implementation requires the definition of a DSM language and code generator by the expert. It makes most sense in cases of repetitive development where it allows making optimal use of the development platform. DSM thus requires an investment of development resources and time to set up the DSM environment. Although this often clashes with the urgency of current development projects the improved productivity is often worth it.

Industrial experiences of DSM consistently report productivity being between 5 and 10 times higher than with current development approaches. Additionally, one has to bear in mind that it is better that the expert formalizes development practices once, and other developers’ generated code automatically follow them, rather than having all developers try to manually follow best practices all the time.

Tools for DSM Implementation

A software development method is of little value if it has no tool support. Designs in a DSM language on a whiteboard do not generate any code, and thus while they help in defining the functionality of the application or system, they do not improve productivity much. DSM has been around for quite some time but companies that chose to adopt it often had to resort to building their own environments to support their DSM language and generators. The huge investment in man years required to do this often led to the decision to look for other ways to improve productivity.

Today, increased competition requires companies to reducing development cost and time to market. Improving the productivity of software development teams has become an important factor in achieving this. After realizing that UML in its standard form does not offer the possibility to significantly improve productivity, companies like my employer, MetaCase (MetaEdit+), along with Microsoft (Software Factories and DSL Tools), Eclipse (EMF & GEF), and Xactium (XMF-Mosaic), have taken steps toward allowing users to build domain-specific model-based code generation environments.

Some do this by allowing the user to make UML more domain-specific by adding profiles to it, while others offer complete freedom in designing the DSM language and code generator. The effort that is required to do this differs from tool to tool. The ideal of course is a tool that gives plenty of freedom to truly raise the abstraction level of the language beyond code and make it as domain-specific as is needed, where more is usually better. Furthermore, the code generator should be fully open to customization, allowing the generation of code that looks like the best previous hand-written code.

In both of these areas, the tool should allow the expert to produce the desired results with as little effort as possible. In particular, the initial definition process should be as efficient as possible, allowing the expert to try out various language approaches quickly. Later on, when the language has been taken into use and real models exist, the tool should handle the evolution of the modeling language and automatically update models as much as possible.

How Does DSM Differ from MDA?

Currently, the Object Management Group (OMG) is intensively promoting its Model-Driven Architecture (MDA), a model-driven development method that comes down to transforming UML models on a higher level of abstraction into UML models on a lower level of abstraction.

Normally there are two levels, platform-independent models (PIMs) and platform-specific models (PSMs). These PIMs and PSMs are plain UML and thus offer no raise in abstraction. It is important here to distinguish between software platforms (what the OMG means with the word platform) like .NET, J2EE or CORBA, and the "product platforms" that I described earlier, which would be at least one level of abstraction higher.

In MDA, at each stage you edit the models in more detail, reverse and round-trip engineer this and in the end you generate substantial code from the final model. The aim the OMG has with MDA is to achieve the ability to use the same PIM on different software platforms and to standardize all translations and model formats so that models become portable between tools from different vendors. Achieving this is very ambitious but also still many years away. This focus however clearly defines the difference between DSM and MDA, and answers the question of when each should be applied.

DSM requires domain expertise, a capability a company can achieve only when continuously working in the same problem domain. These are typically product or system development houses more than project houses. Here platform independence is not an urgent need, although it can be easily achieved with DSM by having different code generators for different software and/or product platforms. Instead, the main focus of DSM is to significantly improve developer productivity.

With MDA, the OMG does not focus on using DSM languages but on generic UML, their own standard modeling language. It is not looking to encapsulate the domain expertise that may exist in a company but assumes this expertise is not present or not relevant. It seems therefore that MDA, if and when the OMG finally achieves the goals it has set for it, would be suitable for systems or application integration projects.

MDA requires a profound knowledge of its methodology, something which is external to the company and has to be gained by experience. Thus whereas the domain expertise needed for DSM is already available and applied in an organization, MDA expertise has to be gained or purchased from outside. In these situations the choice between MDA and DSM is often clear.
Conclusion

Throughout the history of software development developers have always sought to improve productivity by improving abstraction, automation, and visualization. Domain-Specific Modeling combines these methods and copies the fundamental idea that has made compilers so successful: When you generate code, the process has to be complete.

These benefits present a strong case for the use of DSM but at the same time form an important obstacle: DSM implementation requires companies to define and maintain their own domain-specific modeling language and generators. A range of new tools are already helping to make this easier, allowing expert developers to encapsulate their expertise and make work easier, faster and more fun for the rest.

Automating Software Development Processes

Automating repetitive procedures can provide real value to software development projects. In this article, we will explore the value of and barriers to automation and provide some guidance for automating aspects of the development process.

Although few experienced developers and project managers would argue the merits of automating development and testing procedures, when push comes to shove many teams place a low priority on implementing automated processes. The result is usually that, if automation is considered at all, it is given lip service early in the project life cycle, but falls quickly by the wayside.

Experience teaches us over and over again that trying to run a "simple" project by implementing a series of "simple" manual protocols, backed by "simple" written (sometimes, even just verbal) instructions, just doesn’t work well. Even so, many of us still tend to allow ourselves to start the next project with the thought that the manual, protocol-based method will "do just fine."

After all, aren’t we all professionals? Can’t we read a set of simple instructions and just be disciplined enough to follow those instructions when the time is right? Isn’t it a waste of time and money to invest in automating procedures for such a small project? The development team is only a half-dozen people after all—and the argument against automation goes on and on.

If you’re a developer or tester who enjoys spending your time actually adding value to your project, rather than repeating the same routine tasks over and over, you’ll want to consider advocating the concept of automation to your team (especially, to your project manager). If you’re a project manager who’s committed to maximizing the talents and time of the members of your technical team, as well as minimizing the risk of your project failing to deliver on time and on quality, you will want to encourage your team to invest the necessary time and effort required to automate the types of tasks that will be identified in this article.
Why Should I Automate?

You may already be familiar with many of the benefits of automating development processes. Some of the more commonly cited ones are:

Repeatability. Scripts can be repeated, and, unless your computer is having a particularly bad day, you can be reasonably certain that the same instructions will be executed in the same order each time the same script is run.

Reliability. Scripts reduce chances for human error.

Efficiency. Automated tasks will often be faster than the same task performed manually. (Some people might question whether gains in efficiency are typical, noting that they have worked on projects where, in their view, trying to automate tasks actually cost the project more time than it saved. Depending on the situation, this may be a real concern. In addition, automation might have been implemented poorly or carried too far on some projects—but keep reading for more on what to automate, and when.)

Testing. Scripted processes undergo testing throughout the development cycle, in much the same way the system code does. This greatly improves chances for successful process execution as the project progresses. Automated scripts eventually represent a mature, proven set of repeatable processes.

Versioning. Scripts are artifacts that can be placed under version control. With manual processes, the only artifacts that can be versioned and tracked are procedure documents. Versioning of human beings—the other factor in the manual process equation—is unfortunately not supported by your typical source control system.

Leverage. Another big benefit to automating is that developers and testers can focus on the areas where they add real value to a project—developing and testing new code and features—instead of worrying about the underlying development infrastructure issues.

For example, instead of requiring everyone to become intimately familiar with all the little nuances of the build procedure, you can have one person focus on automating the build and have that person provide the team with a greatly simplified method of building, hopefully as simple as running a command or two. Less time spent on builds leaves more time for the tasks that add the most value to the project.

What Should I Automate?

If you’re convinced that automating your development processes is a good idea, the next logical question is which processes should be automated. While the answer, to some extent, is different for every project, there are some obvious ones, as well as some general guidelines that I’d like to offer. Some of the typical targets for automation are:

* Build and deployment of the system under design.
* Unit test execution and report generation.
* Code coverage report generation.
* Functional test execution and report generation.
* Load test execution and report generation.
* Code quality metrics report generation.
* Coding conventions report generation.

The above list is obviously not exhaustive, and every project has its own unique characteristics. Here’s a general, and perhaps obvious, rule of thumb to help identify any process that should be considered for automation: Consider automating processes that you expect to be repeated frequently throughout a system’s life cycle. The more often the procedures will be repeated, the higher the value of automating them.

Once a process has been identified, spend a little time investigating how you might be able to automate the process, including researching tools that could assist with automation, and estimating the level of effort required to implement the automation versus the total cost and risk of requiring team members to manually perform the procedures. As with any other business decision, it really should come down to a cost versus benefit analysis.

You probably noticed that the term "report generation" appears in the above list of automation candidates. The repetition points out another important aspect of automating development processes: the end result of every automated process execution should be a report that is easily interpreted by the team. Ideally, such reports will focus on making anomalous conditions (for example, test failures) obvious at a glance. Also, these reports should be readily accessible to the appropriate team members.

In many situations, it’s even a good idea to "push" reports to the team (perhaps via email or RSS), instead of requiring people to remember to go out and search for them. The basic idea is that the development team should be notified as soon as possible when problems are introduced to the system or environment. A side benefit is that management is provided a more tangible real-time view into the progress and health of the project than is possible with team status reports alone.

When Should I Automate?

Consider automation as soon as you realize that team members are starting to execute a process that meets the criteria discussed in the previous section. For example, automating the build process when the project is nearly over provides very little benefit. It can, however, save dozens, if not hundreds of hours when automated as soon as development begins.

However, before you go off and start automating everything, one caution: be reasonably certain that the procedure will be required for your project, that it will be repeated more than two or three times during the project’s life cycle, and that you understand the steps that need to be executed for the procedure. There are some things you just don’t do very often, and in these cases the costs may outweigh the benefits.

Even for processes where automation is warranted, if you’re truly guessing at the steps involved, there’s a high likelihood that you’ll end up re-writing the whole thing. Re-writing is much different than fine-tuning or refactoring the automated scripts. Refactoring is expected, but scrapping and starting over again means you tried to automate before you understood the process you were trying to automate.

Another danger is allowing your project to become the "proving ground" for automation for the entire organization. If your organization doesn’t already have the infrastructure in place to support development automation, you should provide what makes sense for your project, but try not to allow your team to lose sight of your project’s goals. A project team whose goal is to deliver working software is in no position to develop enterprise-wide strategies and tools. Trying to do so is asking for trouble.

If outside entities begin to get involved in these efforts, I’d suggest that you recommend that a separate project be undertaken to work out the automation infrastructure for the enterprise. Members of this "automation project" team are free to review your project’s implementation for ideas or for use as a launching point. Once this infrastructure is in place, the question of When to automate? is easier to answer for future projects, and less costly.

Obstacles to Automation

If there are so many benefits to automation, why don’t we see more of it on software projects? All software development teams struggle to balance the need to show immediate results with the long-term goals of the project. There are many obstacles to implementation of development process automation. Here are a few of the more common ones:

* Stakeholder pressure. Faced with the desire to show early and rapid progress, teams often overlook or choose to ignore practices that do not seem, at first glance, to directly contribute to getting working code up and running as quickly as possible.
* Focus on core requirements. The team has an overwhelming desire to begin producing working code and tests as soon as possible. Writing code that can be demonstrated is much more satisfying to developers than writing automation scripts.
* Lack of management support. Management may not have a good understanding of automation—how it works and/or the costs and benefits.
* Lack of organizational support. There is no enterprise-wide policy or infrastructure for development automation (standards, tools, expertise, etc.)
* Lack of follow through. Even with the best of intentions, teams can quickly lose their commitment to plans for implementing automated processes.

As with any fundamental change, there must be someone who’s both committed to the concept and who has the authority to make sure that teams follow through with the plan. Not following through is much worse than never having taken the time to investigate and discuss automation. First of all, there are probably other problems that you could have successfully solved using the time and resources and, second, if you fail to implement any automation, when the idea surfaces again people in the organization will point out that it didn’t work the "last time we tried it".
Selling Automation

As implied in the preceding section, the primary obstacle to automation is short-term thinking. Therefore, your primary task is to get your team and management to pause to think about overall project costs and schedule, instead of focusing solely on meeting next week’s promised deliverables. The message should be clear. Meeting milestones is critical, but it’s very possible that you could meet that next milestone to the detriment of the project. But before you try to convince people who are under a great deal of pressure that it sometimes makes sense to slow down momentarily in order to improve the chances for success, you’d better have more than a set of maxims at hand.

This is where Return On Investment (ROI) can help. Most people in the business world, regardless of field, have at least a basic understanding of ROI and agree that, when based on valid assumptions, it can be one of the best mechanisms for evaluating whether or not to take action or adopt a particular solution. The process of calculating ROI is well beyond the scope of this series of articles. However, the Suggested Reading section at the end of the article provides a link to an outstanding article from Edward Adams that describes how to do this, as well as some additional advice on selling the case for test automation, in a very straightforward and easy to understand way.

Although many managers really like to see quantifiable benefits, such as ROI, before making major decisions about adding tasks to the project schedule, others have sufficient experience with software development and are just as comfortable with a common-sense explanation of automation’s benefits. With that in mind, rather than going into a full-blown ROI calculation, let’s just take a look at one scenario that most projects will face at some point to see, from a common sense perspective, the likely outcome of a purely manual process versus a process that had been automated early on in the project.

Manual testing approach. Your team is in the final month of a twelve-month project, and the system test team is starting to perform regression testing by manually re-running the entire set of test cases. Some test cases haven’t been executed in several months, because, frankly, the testers have just been spread too thin, and, due to perceived pressure to deliver, management chose to move testers onto other tasks as soon as they were finished wrapping up a set of test cases.

Regular regression testing was put into the schedule early on, but repeating the same tests over and over throughout the project was (whether or not anyone wants to admit it) eventually considered a "nice to have" and definitely much less critical than other, more pressing tasks. Now some of the regression tests are failing and the test team is logging defect reports.

The development team decides to manually re-run the entire unit test suite (which also hasn’t been executed in a few months, because it wasn’t automated) to hopefully zero in on the source of the problem. The developers find two problems as a result of the unit test run: first, apparently there were never any unit tests written for the parts of the system in question; and second, there are other, seemingly non-related unit tests that are failing. To make things worse, the components that are failing were developed by someone who’s no longer with the team. Even if he were available, he’d probably require significant spin-up time to re-acquaint himself with the failing code, since he hasn’t even looked at it for months.

Automated testing approach: Early on in the project, the decision was made to automate at least the following: unit testing, code coverage, and system testing. The same bug that caused the situation above was introduced in month three of the project. The code coverage report that’s automatically generated as part of the daily build clearly indicated to the development team that there were no unit tests for the related components as soon as those components were placed under source control. Therefore, the developer went back and immediately implemented the unit tests.

After implementing the unit tests and working out the bugs that were initially revealed as a result, the developer checks the unit tests into source control. The daily build automatically picks up the new unit tests, the coverage report reflects that the components are now being tested (as well as to what extent they’re being tested), and the unit test report indicates test success or failure each day. As soon as any change is made to the system and placed under source control, the team will become aware of the impact that change has on the overall system, while the changes are still fresh in the minds of the implementer.

The system tester who’s responsible for testing that same set of features now designs his test case, implements the tests, reports any defects he discovers, etc. until he feels that the code and test case are operating as expected. He then uses the test automation tools to record the execution steps for the test case and checks the resulting scripts and files into source control. The next day, the automated test runner picks up his new test case for execution and the results become available to the test team as part of the daily test run. The development and test teams are leveraging the automated tools and processes for proactive detection as problems are introduced.

Without performing a detailed ROI analysis, which of the two above scenarios is intuitively more desirable? Assuming that the exact same bug is introduced into the system at the same time—month three of the project—in each case, which approach do you think will result in the least expense and least risk? Which will result in the least stress to the team and to management?

Start Small

Unless you’re lucky enough to have a team with multiple members who’ve had a good deal of experience automating development processes, you won’t want to try to take the idea of automation to the extreme. Trying to do so with an inexperienced team or a team that’s not completely sold on the idea of automation will likely result in failure, as both expectations and barriers are set high from the start. Pick one or two areas where you believe there is both high potential value for automation and a low risk that the team will fail implementing automation. You might ask yourself the following questions:

* Which processes will likely be exercised the most frequently?
* Does someone on the team have the experience and/or skill set to implement the automated process?

It’s better to succeed in a small way than to fail in a big way. Your successes just might build on themselves, and when you suggest additional areas for automation, either later in the project or on your next project, you may face little or no resistance at all. To reach the ideal takes a long-term plan, management commitment, adequate tools and training, and of course time. If it were easy, everyone would already be doing it.

Conclusion

I’m well aware that there may be other factors that affect the decision of whether and what to automate than those we’ve addressed in this article. For instance, you may find that the only automation tools your organization will approve for your team’s use are beyond your budget or that, due to your project’s special needs, there are more important factors. Maybe you’re a vendor and your marketing department has determined that time-to-market for your first release is more critical than the quality or the long-term success of the current product.

I’m a big believer in the "it’s all about trade offs" principle. To gain one advantage, you typically accept other disadvantages, and in the end you should try to make decisions that tip the scales most in favor of the "advantages." I would say, however, that if your project is truly one of those projects that is better off not automating, you’re in the minority category—the exception, not the rule. If the primary obstacle to adoption of automation for your organization is lack or expense of tools, there are several free open source tools that can go a long way towards solving that problem. The time you spend investigating automation strategies and tools could very well make the difference in whether your projects successfully meet schedule and quality goals.

.NET Exception Handling

Proper error handling is the bane of the developer’s existence, for errors are both visible and marginal. Errors are visible in that the user sees them. They are marginal in that they are part of "infrastructure;" error handling is a technical problem, and not a business priority.

The marginality of error handling means that the "business case" for error handling is weak, since a heavy duty error handling scheme is to a manager with an MBA from a good school, a market failure.

Unlike programmers, people who are fiduciaries of companies—that is, people with direct responsibility to the bottom line—generally seek ways to quantify risk, and this is usually using an insurance or hedging model. They don’t like to see elaborate error handling in cases where this cost hasn’t been externalized into a customer demand for elaborate error handling or even a software risk insurance policy which states, "we underwrite on condition your programmers do thus and so in error handling."

But this externalization of costs leads to a further problem: as soon as the business requirements get spelled out in detail the question arises to why they need to be in effect rewritten as code. This, in turn, takes us back to the central issue of programming: its necessary marginality as a form of writing, supplementary to the specification.

But that is a large theme; let us focus on error handling.

In my view, a system-level error handling scheme in .NET must meet the following criteria:

* To leave error logging, notification, and display to the highest level tier.
* To avoid elaborate recursive error handling of errors within the error handling, a tricky and in some cases desperate business.
* To add value to the error message where possible, and to Throw or otherwise pass the error message to the next higher level of error handling. To never "lock down" the code to any one vision of error handling.
* To add value in part by conscientiously decorating the message with highly structured information, including the date, the time, the class from where the error emerged, the object instance from where the error emerged, and the type of the error (see below).
* To provide at least the option to write the decorated error to a system log.
* To provide help in two forms
o Help help (general information)
o Action help (information as to what the object is going to do about the error)
* To force the error caller to think about the type of error:
o Pure information that is not, in fact, an error, but which "zero case" the error handles as a professional courtesy...while making sure the error viewer knows that the "error" is not an error
o Warnings, veiled threats, and words to the wise: any situation in which processing can continue without error and in which the object issuing the error hasn’t been compromised or damaged
o Serious errors: processing cannot continue without error. These seem to consist of User errors, which can be recovered from in the sense they don’t affect the integrity of the object.

+ Resource errors, which while not compromising the integrity of the object, prevent it from doing its job in responding to current and future events.

o Critical errors: errors where the software discovers a program bug. For example, the default "not supposed to happen" case in a select case statement should raise this type of error. The object should put itself out of commission.

I have created a demonstration project that illustrates these principles in a .NET-based exception handling framework. After a walk-through of the demonstration/test program I will conclude with an explanation of the underlying code. The entire project, including the test program and a reusable .NET error handling DLL, can be downloaded from this link (zip file):
Trying the errorHandler out

In the shipped code itemized in the next section, run errorHandlerTest.EXE to see the following screen:

In the Level group box, click all the radio buttons to see the result of the errorLevel2Explanation() procedure (see documentation later in this article) refresh the screen with the explanation of each level.

When you’re done exploring, leave the level at "Critical error." Fill in values as follows:

Click Test. Since errorHandlerTest places the call to errorHandler in a Try..Catch block, the Catch intercepts the error that errorHandler Throws to the next higher level along with a vanilla Exception. Since errorHandlerTest is the highest level of error handling and it "knows" it is a Windows application it makes sense to allow errorHandlerTest to report the error using a Windows message box. Here is what should appear:

The complete error message consists of several lines:

* The first line starts with System.Exception because the top level test form has used Catch objException As Exception to Catch the error, and has simply displayed objException.ToString(): but because correspondingly, errorHandler created a New exception with a string set to the full error message, the text of the Exception.ToString() is what errorHandler has built
* Therefore, System.Exception is followed by the meaning of the radio button selected as the Level of the error: criticalLevel.
* This is followed by the source of the error: myObject.myProcedure().
* This is followed by the date, time, and the body of the error message.
* A line break is followed by the full help advice which consists of help text, the errorLevel2Explanation explanation of the error, a blank line, and finally the action you identified.
The blank line appears because errorHandler accesses Microsoft.VisualBasic.Err.Number and Err.Description. If Err.Number is zero and Err.Description is a null string, a blank line appears, otherwise these values are displayed.
You may want to remove the code that displays Err.Number and Err.Description. This is because using them increases your dependence on Microsoft.VisualBasic as a library. This library is full of "legacy" code which doesn’t in all cases work smoothly in Web Services and as a general rule, VB.NET code should migrate away from it.
* The help advice lines are followed by the rest of the Exception.ToString() text, which provide the source of the error.

Take a look at the Event log. Right click My Computer on the Windows desktop, click Manage, open System Tools and Event Viewer if necessary under Computer Management, and open the most recent Application event: this will have the Source set to errorHandler Test:

As shown above, because strEventLog was included, an event was added to the Application log.

Try the errorHandler out to see if it meets your needs and that of your team.
Additional Code Explanation
Project Structure

The following directories and files are available, containing the errorHandler and its test application.

* ErrorHandler: directory containing all files needed
* ErrorHandler/AssemblyInfo.vb: assembly information for errorHandler.DLL
* ErrorHandler.SLN: solution file including both errorHandler and errorHandlerTest
* ErrorHandler.PROJ: project file for errorHandler.DLL
* ErrorHandler.VB: Visual Basic .Net source code for errorHandler.DLL
* Bin: directory containing errorHandler.DLL
* ErrorHandlerTest: directory containing the errorHandler test application, containing ErrorHandlerTest.SLN, ErrorHandlerTest.PROJ, and Form1.VB.

The errorHandler Class

errorHandler is a stateless .NET class which exposes two Shared methods: errorHandler() and errorLevel2Explanation():

errorHandler()

errorHandler(strMessage As String,
ByVal enuLevel As ENUerrorHandlerLevel,
ByVal strObject As String,
ByVal strProcedure As String,
ByVal strHelp As String,
ByVal strAction As String,
ByVal strEventLog As String)

strMessage is the only required parameter of errorHandler() and it should describe the message. There’s an art to composing an error message, of course, and errorHandler() can’t prevent you from composing a bad one. In general the message should avoid blaming the user who unlike the programmer is usually scared of error messages.

The remaining parameters are all optional in the sense of overloads-optional.

* enuLevel is the error level as described above.
* criticalLevel: a programming error has occurred.
* informationLevel: not an error: but we need to provide some information using the error handling system.
* resourceLevel: an object cannot get the resources it needs from the system to operate properly. In .Net, for example, this would be any failure to create a reference object (other than Boolean, Byte, Short, Integer, Long, Single, Double or String), which needs storage in the .Net heap.
* userLevel: the object user has supplied invalid data, typically as a procedure parameter, such that the request made cannot be honored, but the object itself is OK.
* warningLevel: there might be a problem, but probably isn’t.

strObject should identify the object producing the error...not its class. Ideally, you’ve named each class instance using a method I will fully describe in a subsequent article, where the name includes the class name, a sequence number, and the date and time. However, something like classname instance is perfectly acceptable for short-term use. It identifies the class, but makes it clear that an instance produced the error. strObject defaults to "Unidentified object."

strProcedure should identify the "procedure" which in VB.Net identifies either the method or property, generating the error. strProcedure defaults to "Unidentified procedure."

strHelp can be a long or short string with long or brief help information. There doesn’t seem to be much point in subordinating an elaborate "help" system to the errorHandler which to be reusable interface needs to be light in weight and in spirit, and you can, if you like, get the help information from another subsystem and pass it, if it is of reasonable size, as a string. strHelp defaults to a null string because it looks unprofessional to default to "Help is not available." There’s a public service ad on Hong Kong TV which advises people working in shops never to say "no" in such a way as to make the customer feel as if she’s undeserving of help.

strAction should identify what is being done in response to the error condition. It’s a separate consideration from help information. It defaults to a null string.

strEventLog when present and not null causes the error to be logged (with its full "decoration" including date and time) using System.Diagnostics.EventLog.WriteEntry() with two parameters: strEventLog and the fully decorated message.

On a Windows system, this adds the message with decoration and new-lines to the Application log which can be viewed by right-clicking My Computer on the desktop, opening Manage, opening System Tools and Event Viewer and clicking Application.

errorLevel2Explanation()

This shared procedure accepts an enuLevel enumerator (of type ENUerrorHandlerLevel) and returns its explanation:

* criticalLevel: "Programming error" is returned
* informationLevel: "Not an error, information instead" is returned
* resourceLevel: "System not able to get a needed resource" is returned
* warningLevel: "Warning, processing continues: there may be a problem" is returned
* userLevel: "Have recovered from a user error" is returned

One problem throughout this short and simple class is its exclusive use of English, but for ease of modification, the error level explanations are symbol constants, and not strings.

Best Practices for Object/Relational Mapping and Persistence APIs

Over the last decade there has been a lot of effort put into object/relational mapping, which refers to techniques for resolving the mismatches between the object-oriented world, with its encapsulation of data and behavior, and the relational world, with its tables and columns. There’s not only a difference in terms of data types (and complexity of these data types) but also in terms of relationship types. The object world has a variety of relationships (aggregation, composition, association, inheritance) which cannot be mapped directly to the database world.

The general topic of object/relational (O/R) mapping can be divided into two areas of concern: the mapping itself and the persistence API. The persistence API not only acts as an indirection layer for the database, but also hides the mechanics of mapping the objects to the database tables. A good persistence API should do this while not constraining the object modeling in terms of data types and relationships.

In this article I will begin with a discussion of home-grown vs. off-the-shelf persistence solutions, including areas to consider when deciding between the two, and advice for choosing the best off-the-shelf solution to meet your needs. I will also share suggestions and advice from my own experiences with O/R mapping and persistence APIs. It is not my intention to explain all of the background details of these topics, but to focus on "best practices."

Home-Grown Mapping


I have been directly involved with both home-grown (custom built) and commercial-off-the-shelf (COTS) mappers, and have also observed several other home-grown mapper implementations, each approaching the subject in a specific way. In my experience, the primary drawback of "rolling your own" persistence mapping implementation is that limited resources do not allow for enough time to think everything through, to improve the framework over time, and to backtrack if you realize that design changes are needed.

Because an O/R mapper is a generic piece of software, it is typically hard to explicitly list which aspects are of the most importance to you (and if you build your own, you will not be able to focus on all of them). I do not mean to say that you could not envision a good design, but that it would take a lot of time and effort to fully implement a solution that meets all of your needs.

I observed the following limitations in just one home-grown O/R mapper:

* It provided no support for association relationships; only containment relationships were supported. This was a serious constraint when defining the object model.
* It offered no support for transactions.
* It only supported one RDBMS.
* The API was not type safe, which caused a lot of errors that could only be detected at run time.
* Testing of the O/R mapper was underestimated. Not only were there a lot of possible paths through the code, there were also stability and performance issues to be considered.
* The designer had to create and maintain the object model by editing a plain text file without any special editor. I know the saying "a fool with a tool is still a fool," but not being able to visualize one’s own object model is like walking around in the dark—you don’t see where you are heading and you have difficulties in pointing the others in the correct direction. In this case, the object model, which was like a red ribbon winding through the whole architecture, could not be clearly communicated to the team.

I hope these examples are enough to help you to avoid stepping into a home-grown solution. Of course, a home grown O/R mapping project would be fun to do from a development point of view, but a COTS (commercial-off-the-shelf) O/R mapper will be cheaper—unless you consider O/R mapping as one of the core competencies of your business.

Selecting a COTS O/R Mapper

Below I list some criteria you might want to consider when selecting an O/R mapper.

* Consider whether the tool will restrict your modeling freedom too much. For example, many tools don’t support relationships on abstract classes. A workaround for this is to duplicate relationships on concrete classes, which is less ‘OO’, but works for these tools.
* Consider whether the O/R mapper that allows you to model visually (preferably using UML).
* If UML is important to you, ensure that you can either import UML into the O/R mapper, or that you can export UML from the O/R mapper.
* Take a close look at the programming model the O/R mapper imposes and see whether it is compatible with the things you want to get out of it.
* Look at the range of mapping possibilities to ensure that the kinds of relationships you envision between your objects and tables will be supported. Typically, most O/R mappers support a large range of features, but not every mapper supports every type of relationship.
* Assess the performance, even if you think you do not have a lot of performance demands. Testing the performance can also give you a chance to learn and assess the API.
* If you prefer or need to start with an existing database schema and then map objects onto it, assess whether the O/R mapper supports the database system you want to use.

Selecting a Persistence API

The O/R mapping features are only part of the story. The other part of the story is the selection of a good API for persisting objects, and this part has a lot more visibility to your development team than the O/R mapping part. While the O/R mapping functionality will only be exposed to a few team members who are dedicated to maintaining the persistence layer, the persistence API defines the interface that the whole development team will use.

Persistence APIs can be divided into two categories: transparent and non-transparent.
Transparent Persistence APIs

A transparent persistence API hides the persistence completely. A transparent API does not need to have a lot of methods; a load and a save method is sufficient most of the time. Typically, a lot is defined declaratively instead of procedurally. Hibernate and JDO are examples of transparent persistence APIs. Let me clarify with an example:

An Insurance object can contain 0-n Warranty objects. The client application updates an attribute of an Insurance object. Semantically a Warranty is contained by an Insurance, so when you update an Insurance it is possible that you implicitly update its Warranties. According to the requirements, this might be a correct design, but it could have a negative impact on performance, especially if I am not aware that the implicit update of the Warranty objects is happening. When I only have modified an attribute of the Insurance object, I should be able to limit the persistence manager to this functionality.

The beauty of this is the "magic" way in which the persistence manager knows what to do. The negative side is that the persistence manager is thinking in your place.
Non-Transparent Persistence APIs

A non-transparent persistence API has a lot less "magic" inside of it. When compared to a transparent persistence API, it has a rich API, offering a lot of control to the user of the API.

Consider a transparent persistence API with a single method Persist(). This persistence method does all the magic behind the scenes, like checking whether there are associations that potentially need to be persisted as well. Although this might sound attractive, when selecting a persistence API, ensure that you can optimize. When touching associations, it is possible that the associated objects haven’t been persisted yet. What should the Persist() method do? Persist those objects first? I say that it is better to put the client in control.

To illustrate the power of a non-transparent persistence API, I’ll use the SimpleORM API as an example. Consider an insurance class with two methods. The first method will explicitly load all of the children linked to this object:

insurance.getAllChildren(insurance.Warranties)

The second method will only list those items that were added to the object in memory and the ones that are already retrieved from the database:

insurance.getRetrievedChildren(insurance.Warranties)

The advantage here is that the user of the API can "see" what he is doing, and can make better decisions regarding performance costs. (Also during code reviews this visibility can make life a lot easier.) In contrast, due to the fact that a transparent persistence API has a very generic interface (e.g. Save() and Load()), it also creates the illusion that database actions are cheap.

The cost for the non-transparent API is that the interface is more complex than a transparent persistence API. However, if you are planning to write your own persistence API, I recommend a non-transparent persistence API.

It is not my intention to classify all transparent persistence APIs as "do not use." But if you are considering a transparent persistence API, I would advise you to assess its performance carefully.

In case you are not satisfied with the persistence API that is provided to you, you can also decide to wrap it such that it maps onto your needs. In the next section, I’ll elaborate on some good reasons to wrap a COTS persistence API.
Wrapping a COTS O/R Mapper’s Persistence API

After you have selected a COTS O/R mapper, you should also decide whether to use its persistence API directly or to wrap it. Good reasons for wrapping the O/R mapper’s persistence API are:

* You want to add some extra logic (for example, field validation is not part of the O/R mapper, but you want it to be an inherent part of the persistent objects).
* Some O/R mappers generate code for the persistent objects, others expose a generic API. Typically, the generated persistent objects are type safe, while the generic API is not. Type safety is a good reason for wrapping the O/R mapper’s persistence API.
* Apply the subsystem principle: you want to avoid a tight coupling with a specific O/R mapper. Therefore you treat it as a subsystem and work interface-based.
* You want to limit the features that your clients can use. For example, your mapper might support the use of direct SQL while you don’t want to expose such a feature.
* You might want to expose certain services in a different way. For example, you might want to introduce a query object rather than to expose an OQL query interface.

Of course you need to place everything in perspective: if you are creating a throw away application, you don’t need to worry about aspects such as maintainability, extensibility, and resilience. For strategic applications however (commercial software products, product families, core business applications, etc.) you really should spend some time evaluating the pros and cons of different approaches.

A bad reason to wrap a persistence API is that you think that you will be able to boost performance. In general, you won’t be able to do this because the performance is inherent to the internal design of the COTS component. Only in rare occasions you will be able to turn the performance to your advantage by wrapping the COTS component.

A better way to approach performance is to invest some time in seeking a usage pattern that is optimal for your situation.

Wrapping and Object Management

Another way to discriminate between persistence APIs is the way objects are managed.

In one technique, the persistence manager is an object factory (using the factory pattern):

MyPersistenceManager mgr = MyPersistenceManager().Instance;

Insurance insuranceObj = mgr.CreateInsurance();

mgr.Persist(insuranceObj);

An advantage of this approach is that the manager always knows the current state of the object (new, retrieved, already saved). The manager can use this information to its advantage, resulting in good performance.

In other APIs, such as JDO, the objects are not created within the context of the persistence manager. The persistence manager takes an object in and determines what to do with it. In this JDO example, the manager (mgr) does not know the current state of the insurance object (unless perhaps it caches the information):

PersistenceManager mgr =
persistencemanagerFactory.getPersistenceManager();

Insurance insurance = new Insurance();

insurance.SetPolicyNumber("2005001001-110");

mgr.makePersistent(insurance);

Okay, by now I hear you saying "All of this is very interesting, but why is it of my concern?" Two reasons:

First, consider object management when you want to wrap the persistence API. Performing the wrapping without knowledge of the pros and cons of different persistence API approaches would be very dangerous.

Second, consider the performance implications. In the case of the object factory mechanism, you can be quite sure of decent performance. In the latter mechanism, it’s advisable to assess the performance by means of some unit tests or architectural prototypes.

Disconnected Objects

A typical disconnected object scenario starts with the retrieval of the object from the database in the server app, then streaming it to the client app, then in-memory modification on the client app, and finally streaming it back and storing it to the database again in the server app. The way the persistence framework handles disconnected objects has quite a big impact on the performance.

One way to improve the performance of disconnected object persistence is through merging, as in this JBoss example:

Insurance insurance = Util.deserialize(input);

// Changes are monitored as of now.
entityManager.merge(insurance);

insurance.setSubscriptionDate(d);

entityManager.persist(insurance);

In the above example, only the change of the subscription date is stored in the DB.

Another area of performance concern is keeping track of changes vs. figuring out at the time of persistence what changes have occurred. The latter technique is more dangerous for performance, especially when complete object trees are being saved. The algorithm to find out the changes has to be fast, or the original data needs to be cached in order to achieve a good performance.
O/R Mapping Best Practices

I have distilled the following "best practices" from my experiences:

* Don’t work against the O/R mapper’s persistence design. Rather, take the O/R mapper’s design principles as a constraint and exploit them. If you don’t, you’ll have to pay in terms of efficiency, performance, etc. Also, make sure that you know the basic concepts of the O/R mapper’s design.
* Wrap the O/R persistence API and treat it as a subsystem, such that you work interface-based, which eases the prospect of switching later. I don’t say that it won’t hurt to switch to another O/R mapper, but at least the pain can be isolated.
* Check the querying capabilities of the O/R mapper’s persistence interface. Especially, check whether aggregate functions can be used and whether you can query for raw values rather than plain objects. Objects can be too much of a good thing (object bloat) when, for instance, you just need a couple of values to fill a grid.
* Implement field metadata wisely. Generate field metadata (such as size, etc.) in-line in metadata classes instead of using reflection. This improves performance, and also makes it easy to debug the code.
* Put field validation at the level of the metadata. Make sure to expose your field-level business rules such that you don’t require a round-trip from your client to your server application in order to know whether or not the object is in a correct state to persist it. Tools such as SimpleORM suffer from this (typically field validation is foreseen internally, but not exposed).
* Be careful when calling a field a "mandatory" field. Typically, O/R mappers foresee the ability to tag certain fields as mandatory. In my experience, I have seen that this construct is used too much. Typically fields are mandatory dependent on the state of the object and such interdependencies can seldom be expressed. Therefore, the best practice is to limit the mandatory fields only to those fields that make the object incorrect within the application domain.
* When defining the persistence API, guard its consistency and ease of use. When specifying the interface of your persistence API, make sure that the parameters of each persistence method are consistent with the rest of the interface. As an example, the following dummy interface is not consistent because in one case an enumeration is used (RelationshipName), and in the other case a string:

SaveRelation(RelationshipName name, object value)
SaveAttributeToObject(string attributeName, object o)

* I would suggest to either go for a type-safe approach or for a generic approach, but not to mix them in one interface. If you want both of them, then put the generic methods on a separate, more generic interface. Don’t make the interface more complex than it should be.

* Make sure that the API can be used in a way that a client application can take advantage of its knowledge to optimize performance.

What Is A Professional Programmer?

How do people become professional programmers? Many people go the "traditional" path through a computer science or software engineering education and from there into professional programming work.

Others become professional programmers by accident. A person writes a small program to help at work, and their workmates say, "Oh great, you can write programs! You're our programmer now!"

Other people start out as hobbyists and follow a less traditional path, not always getting a degree, but clearly wanting to be programmers from the start and working actively towards that goal.

I've been a hobbyist programmer since I was 6. I wasn't writing anything amazing back then but I had started writing and soon found it was absorbing most of my time. Since I never really stopped, that gives me 24 years "programming experience" and counting.

At first I was into writing computer games. Later people asked me to write programs for them, and sometimes I even got paid. From this I learned that software is always for something. Programs are not self contained worlds of their own. People expect things out of a program that have more to do with Japanese or Geophysics or Engineering (or whatever they've got in mind) than with how a computer works. I had to learn something about all those domains in order to write programs for them.

At university it didn't take long before I was a tutor, and that's where I found I enjoy teaching, and especially enjoy teaching programming.

While I was at university I got my first "real" job, writing Visual C++ code for a financial database company. In terms of design and theory it was lightweight stuff. But in terms of working with others on a large project I was being thrown in the deep end! They had gigabytes of source code, growing cancerously through the efforts of a dozen developers of wildly differing skill levels.

In spite of my programming skills being well above average there, I learned to settle for being a junior programmer, a little fish in a large pond.

Skipping along a few more jobs and a lot more years, today I am a senior developer in a small research group—a big fish in a little pond. I've had to teach my co-workers a lot about professional programming, because most of them haven't been in industry to get that taste of what large code bases and diverse skill levels do to programs if you aren't using those "professional" skills to keep everyone pointed in the same direction.

There's quite a gap between "being able to program" and being a "professional programmer." It took me 15 years to go from beginner to hotshot programmer, then another 10 years to go from hotshot to professional—and I'm still learning.

Whatever the path we follow, most professional programmers have in common the fact that they learned to code first and how to be a professional later.
The Meaning of "Professional"

So what does it mean to be a professional programmer? What does it mean to be a professional anything? Some definitions simply say to be a professional is "to make money from a skill," but true professionals also have a set of qualities often described as "professionalism." In my opinion, these qualities are: trustworthiness, teamwork, leadership, communication, constant updating of skills, an interest in minimizing risks and accountability. Each of these effect the professional programmer in certain ways.

Trustworthiness The concept of trustworthiness applies in several different ways for programmers. Can you be trusted with a job? To perform a task without someone checking up on you? Can you be trusted to ask for help when you need it?

If you're given clients' data or have signed a non-disclosure agreement, then you are being trusted to respect privacy. You are trusted to check license agreements on third party tools or libraries and to get licenses or permission as required. And like any professional you are trusted to simply do a good job.

Teamwork Will you genuinely cooperate with your team mates? Will you work to mutual advantage and not just your own? Can you trust your team to work with you? Can you do your share of the work and trust your team to do the rest? And can you accept your management (and sometimes even clients) as part of the team, everyone trying to get the same job done?

Leadership Showing leadership means both earning respect from others and knowing what to do with it. Recognize the skills of your team members, and make sure you can offer each person challenges and development without exceeding what they can cope with at a given time.

Leadership involves not always getting to do the "fun" parts of a project yourself (that scary "delegation" word). It also involves not asking anyone to do a task that you wouldn't be willing to do yourself. It's not just the managers and lead programmers who need to show leadership, it's any professional programmer. The best programmers to work with are the ones that know what's going on, not just their little tasks.

Communication Respecting the people you work with, and your clients, enough to really listen to them is a critical part of communication. Teamwork can't happen without good communication, nor can accountability.

Communication is critical for helping clients to produce usable specifications and feedback. Will you question whether the specs you are given really will serve the purpose that the client has in mind?

Communication skills help with making meetings timely and effective. A professional's communication is effective and to the point, whether in person, in email, on the phone or in written documents.

Documentation at first seems like a programmer-specific concern until you consider how many people require documentation in a serious project: other programmers need high level, API level and in-code documentation; managers need planning, progress, and bug documentation; lawyers need proof of what was done and when; and users need documentation on how to use the software.

Updating Skills Keeping your skills up to date involves staying aware of what's going on in your industry. What are the current ideas about methodologies like eXtreme Programming? What libraries and tools are out there that might support your project? What are the current refactoring tools? How about standards, file formats and protocols? Are you up to date with Unicode, XML, SQL, and all the other acronyms? Perhaps you're missing out on something if you're not. What platforms are your potential clients using? Should you be learning about cross platform development?

Basically you need to possess a genuine interest in your field, and to read broadly so you know what's out there and which areas to then read deeply about. You also need to accept that even (or should I say "especially") the very best programmers are still learning.

Minimizing Risks Familiarity with best practices, combined with a healthy dose of common sense, will take you a long way towards managing risks. Professional programmers keep track of known bugs or any other change they intend to make. Bugs are risks, and a simple database can prevent you having a product ship with bugs you'd simply forgotten.

Another risk that's often not properly considered is any and all changes to the source code. Source is your livelihood and any change can be a mistake. There's good software out there that will keep track of every revision of your source code and even help merge code that multiple people have changed.

Professional programmers are careful to do enough testing. A software company will generally have testers but the developers need to know how to get the most out of testers and also how to write their own unit and regression tests to make sure every change in behavior is noticed and checked by a human.

Keeping your code simple and well styled is another commonly overlooked way to manage risks. If anyone can look at the code and see right away what it does, you are far less likely to find bugs in it later, and you are less likely to have a junior programmer attempt to change something without understanding it first.

Another risk is the client changing their mind, or more often changing their specifications because they've realized it wasn't what they had in mind. Write your code to be modular and reusable and you won't have any trouble adapting it to changing needs.

Accountability Writing code for others is a responsibility. You need to make sure your software is reliable. You need to make sure you and the client truly understand the requirements and specifications. You need to have documentation of your work, all current and past bugs, your progress, any problems, signed-off milestones, and more. You are also required to know about some basic legal issues, like software licensing, the terms of your employment contract, and intellectual property law.

* * *

As you can see, there is a huge gap between "coding" and "professional programming." Most programming courses focus on the coding side of things, and the professional skills tend to be glossed over or not covered at all. I have found myself regularly teaching these skills to new co-workers, which highlighted the need for "professionalism skills training." Teaching my co-workers reminded me how much I enjoy teaching. I decided to teach more people by trying my hand at professional writing for a change.

I set up a web site, which is completely independent from my day job. The site is called Developing Programmers.com. It is devoted to teaching people how to develop into professional programmers. Since founding the site, I've been presenting the tools and ideas that I think professionals should know about.

Some of my articles simply refer to other sites of benefit to would-be professionals. I research other articles from scratch: tutorials, guides, and discussions of things professionals should be thinking about, like revision control, documentation, keeping your group pointed in the same direction—and of course, each of the aspects of professionalism that I listed earlier.

These days I consider myself to be a professional programmer, though I am still discovering the depth and breadth of what exactly that means. Perhaps that ongoing exploration of programming and of professionalism is what makes this for me a career and not just a job.

Career Paths for Programmers

I recently interviewed for a Business Analyst position with the CIO of a large multi-national software development firm. This man was in charge of the company's worldwide IT operations, including offshore development projects, for which he was searching for qualified Business Analysts. The interview quickly became a casual conversation about current trends within the IT service sector, how the company was planning to take advantage of those trends, and, most importantly, how I could fit into those plans. It was during his evaluation of my skills that I asked how I fit and whether it was technical or business skills that were most valuable to his projects. The CIO summed up his advice about my career path with one small sentence: "Stay on the business side."

Business skills, in this CIO's view, were most important to his future projects and the industry as a whole. His reasoning was that he could train anyone in the technical skills he needed for a project, but finding those people with the necessary business skills to guide an IT project to success was something that could not easily be obtained. He went on to say that he found it difficult to find people who could communicate on even the most basic of levels. I asked if my background as a developer would help in getting a business analyst job, and he conceded that although it's not a requirement, it certainly would help matters as long as I could prove that I wasn't "too technical."

His comments are consistent with the trend that all US-based programmers have observed since the late 1990's: global salary competition amongst programmers, and a growing view in big business of programming as a commodity skill. It's hard to compete with a developer in Russia or India who can work for a fraction of what I make minus benefits. The CIO managed to reaffirm the subtle, but major, shift from technical skills to business-technical skills in today's labor market. I gave weight to his viewpoint since the people in his position are the trendsetters of the technology industry. They are the ones who set the directives for a company's IT needs, and often, the requirements desired for the higher-paying positions.

I did a little research and found that the US Bureau of Labor Statistics Occupational Outlook Handbook predicts that computer systems analysts are expected to be among the fastest growing occupations through 2012. The Handbook describes a systems analyst as someone who may plan and develop new computer systems or devise ways to apply existing systems' resources to additional operations. It describes a computer programmer as someone who writes programs according to the specifications determined by systems analysts. (The book does not separately list business analyst as an occupation.)

According to the Handbook, in the US systems analysts held an astounding 487,000 positions in 2004 (up from 468,000 positions in 2002) compared with 455,000 jobs in 2004 for computer programmers (down from 499,000 in 2002). The Handbook also states that employment for computer programmers is "expected to grow much more slowly than that for other computer specialists." And recent estimates by the Economic Policy Institute have put the number of jobs being offshored at approximately 330,000 to 500,000 jobs. About 100,000 of those were full-time computer programming jobs.

The key to maintaining a good employment outlook in IT, it seems, is to move out of programming and up into more business-oriented IT positions such as systems analyst, business analyst, project manager, or systems architect. However, a computer programmer can't just decide to become a systems analyst or project manager overnight. The journey takes time and requires the right amount of experience and learning to be successful.

Making the Shift

So you've seen the statistics and watched as the jobs in your market slowly disappear. You want to move more to the "business side," but you don't quite know how to do it. As I'll describe next, making the shift can be done on-the-job by gaining more responsibility, polishing up your problem-solving skills, and using creativity in your work.

I began my journey into systems analysis and design by accepting more responsibilities throughout the project I was on when things proved too overwhelming for my superiors. I gradually accepted more of the project management and business analysis responsibilities when the opportunity presented itself. For example, I would walk to Suzy in accounting and work out a new enhancement with her one-on-one rather than wait for my manager to do so. Over time, as my manager's confidence in my abilities grew, these responsibilities became a part of my job. It wasn't long before I became the Programmer Analyst, and ultimately the Project Manager, as new positions were created to fulfill demand for our work.

When the need arises, I recommend walking to the end user yourself and working with her one-on-one. Your manager will be relieved when he discovers that you are capable of communicating with his end-users, identifying their issues, and resolving those issues before they are brought up in the weekly manager's meeting. Even the best IT managers need a subordinate who is visible to the users who they can trust to get the job done. If a manager is slowly factoring himself away from the day-to-day workings of the project, welcome it. The higher visibility that you are obtaining can be translated into higher value—and that can result in a promotion. Over time, your increased interactions with more business-oriented people will make you more sensitive to business concerns.

A good subordinate has to be open-minded and creative. When solving problems, one has to always believe that there is a way to accomplish something, even if it's never been done before. Sometimes, just listening to the user will produce an idea. A lot of issues may come down to the business process that the system is attempting to replicate. I have had users actually solve a business problem for me just by listening to what they had to say!

Whether you're open-minded and creative or not, you can still work towards more business-oriented positions. After all, business systems analysts and project managers are only a small subset of the many positions opening up each year to address the issues of complexity through simplicity. If you love programming, you don't have to necessarily give it up.

Jobs To Pursue

Senior Technical Positions

Developers will often find that they may have to work side-by-side with the users to iron out difficult bugs. It can be difficult, if not impossible, to fix these problems when both parties can't communicate effectively. There was always a time in most of my work situations when the developer had to talk with the users or other developers directly to fix difficult issues. This is the programmer's chance to show management that he or she is someone who can communicate and utilize analysis methodologies—otherwise known as a "programmer analyst." A programmer analyst is also usually someone who has some years of technical experience, and a certain depth of technical knowledge.

Programmers who seek advanced technical skills without too much end-user interaction may find themselves gravitating toward the design & architecture side of the business. Although these types of positions are still relatively technical, they often involve making key decisions to address how the new system will fit into the organization's overall IT plans. In order to be successful, the architect needs to understand and control the elements associated with the utility, cost, and risk factors of the proposed solution.

System architects must make very educated decisions about how to decompose and isolate the different components that will be required, how to fit these components into the existing infrastructure, and in what order to implement each component. It can be a disaster to implement an online ordering system that isn't compatible with the organization's current accounting packages. The architect must identify these types of issues and present them to non-technical management in words they can understand.

Business and Systems Analysts

My job searches have suggested that business and systems analysts with a good programming background and a high-level of "business savvy" are becoming the next hot ticket. More and more organizations are finally hiring business analysts to explore, record, and recommend systems that fit the business—as opposed to the other way around.

The business analyst must often work with project managers, systems architects, and systems analysts, all of which are growing occupations that can make the difference between success and failure. In some cases the business analyst's responsibilities are being combined with that of the systems analyst or the project manager under the guise of "business analyst" or "business systems analyst." A quick search on Dice.com will reveal that many business analyst jobs have hidden deep within their job descriptions requirements to develop technical specifications or to guide and manage projects. My first business analyst job required both project management and systems analyst skills. These positions are sure to become more common as organizations struggle to reduce project failure and development time.

Project Management

According to the Bureau of Labor Statistics' Occupational Handbook, employers prefer project managers who possess advanced technical skills that have been acquired through work experience. The project manager is often responsible for hiring the staff, setting the schedule, and keeping track of the progress through every phase of development. This person is also responsible for assigning the work, dealing with everyday problems affecting that work, and making sure each analyst or programmer is carrying his own weight. The project manager can best carry out this function if he truly understands the work he is managing.

The project manager must also be a "people person" as well as a "technical person" in order to succeed. This individual must work with technical and non-technical staff at every level of the organization in order to succeed in his goals. Additionally, the project manager has to manage his team effectively to produce the desired product on time.

Management

The ultimate assignment for many IT professionals looking to move up the IT food chain is to become the manager. The Occupational Handbook explains that "employment of computer and information systems managers is expected to grow faster than the average for all occupations through the year 2014." These job opportunities are best suited for applicants with computer-related work experience and often require an advanced degree, such as an MBA. And of course, strong communication skills are a requirement for any management job in IT.
Skills To Develop

Okay, so you've heard all about what's required and where IT is going, but how can you capitalize on this new information?

My interview with the CIO and my experience in the field have shown me that companies want IT professionals who can understand what their business is and how to apply technology to make it better. Being able to follow directions is important, but being able to take some initiative and make your own judgments without handholding is equally important. The solution is to differentiate yourself from the traditional developer.

We have already discussed two ways of building up your current skills—acquiring business knowledge and advanced technical knowledge—but two other areas are important as well: communication and leadership.

Whether that CIO I interviewed with believed that communication skills could be learned or not is irrelevant. Everyone can learn to be a better communicator with practice. The difference is that communication skills take much longer to develop. Communication takes the right mix of experience and training to become effective. I have worked on this since my college days and have had great success in my career as a result.

I learned to communicate more effectively by dealing with those who couldn't. Many software users can't understand the technical side enough to describe any of their requirements in any type of detail regardless of their background. On the other hand, many technical people don't understand the intricacies of the business processes they are implementing because they can't openly communicate with the users. Learning to communicate, and having the patience to gain knowledge from the user, is an essential skill that many of my former and current coworkers don't have.

To add to your problem solving skills, instead of asking your superior or a more experienced programmer to help with a problem, take it upon yourself to find the answer to that complex problem. Before too long, you can be the one who others consult when there is a problem to fix or a new project to complete. Gaining problem-solving experience not only improves communication, it also improves your chances of moving into analyst and management positions. Eventually, you can do as I did and get your own project to manage.

The key to moving up the ladder at any company is to let them know what you know. Answer those questions, solve those problems, accept those new projects, and don't be too shy to share a better solution. It could mean the difference between being "just another programmer" or being the top candidate for a promotion.