Software Estimation Using Pattern Analogies

Predicting the future has never been easy. This is particularly true in software projects. No one seems to know how long a project will really take or what the final cost will be. In our team’s effort to overcome the challenges of software cost estimation, I saw that we needed a way to estimate a count for the lines of code (LOC) in each software module. An LOC estimation does not give you the total picture, but it’s an important and useful building block of information in any formal or informal estimation technique. The technique I developed to fill this need is called Pattern Analogies. We used past experience with design patterns (both published and home-grown) as a basis for estimating LOC. Once we had an LOC estimate we could use our choice of a number of models to calculate effort, cost, and duration.

This article is organized as follows: The first section defines what Pattern Analogies are. The second section shows a case study using Pattern Analogies. In the third section, I explain where and how to fit size and effort estimation in common software lifecycles for agile teams. Before concluding, I present some keys for success with this technique.

Pattern Analogies Defined

We developed this system for estimating LOC from Watt S. Humphrey’s "proxy" concept [Humphrey, 1994]. Humphrey estimates size by identifying modules from a design, categorizing each module accordingly to its functional group, and then using the historical size of that group to estimate size. Since our team develops software using patterns, we experimented with exploiting those patterns in estimating size. Once we determine size we use COCOMO II to estimate effort (in man/months) and cost.

We find the lines of code (LOC) for a module by "analogy." This analogy method is simple and does not require calculating geometric distances of other complex statistical values. A traditional analogy size estimation method estimates the size of a unit of software (method, class, system, component) by comparing it to a similar example. Then the software developer assumes the new application will have the same size as the one she is comparing it to. In order for this method to be effective you need to do the comparisons and the most granular level possible.

For example, if you calculate the size of the methods of a class by analogy, the final estimate is going to be more accurate than if you calculate the size of the class as a whole by analogy. Of course, the more granular the unit of software, the more difficult it is to find a something to compare it with. Also, the more granular the unit of software, the more time consuming the estimation process becomes. This is where Pattern Analogies come into play. This article proposes that the most granular software unit to do analogies on is a design pattern.

What we do is divide a module into components, where each component is an instance of a design pattern. It could be a pattern from the "Gang of Four" [Gamma et al, 1995] or other patterns such as Data Access Object (DAO), which encapsulates the access for a database table. (A pattern could also be domain-specific or have been invented in-house.) Then for each pattern we have a rule for how to estimate the LOC for that pattern.

For example, for a DAO we have defined for the languages that we use (Java and PHP) an estimate of how many LOC per database column there are in each DAO. The calculations are based on statistics from past projects. So, if a module is envisioned to include DAO’s we can estimate with a good degree of certainty how many LOC those DAO’s will have.

Another example is estimating LOC for a Gang of Four Mediator. The rule for a mediator is as follows: we decide if the mediator is easy, normal, or complex. Then for each type we have a different factor to multiply against the number of components or widgets that the mediator will be dealing with. We get our LOC estimate by multiplying the factor times the number of widgets.

Note that we also have a rule of how to count the number of widgets. Each widget is a swing component that the mediator is aware off (this means the mediator has access to the widget directly). We calculated the factors by counting the LOC for over 20 mediators from past projects. We also counted the number of widgets each mediator deals with. The last step was to define the mediator as simple, normal, or complex according to the criteria in Figure 1. With that information we simply found the ratio of LOC divided by number of widgets for each mediator. Then we calculated the average of the ratios for all the mediators in the same category (simple, normal, complex).

We also have similar rules for Commands, Command Holders, Visitors, and other internal patterns like Handlers (a Handler is a stateless object that controls access to a DAO for our web services) and Tasks (a series of actions that require monitoring and status reporting). A rule is a table similar to Figure 1 for each of these patterns. For this method to work everybody on the team has to agree to use standard design patterns to develop their part of the application. For the part of the application for which standard design patterns don’t apply (usually a small portion) we ask the person in charge to give us a LOC estimate of it based on her experience.
Using Pattern Analogies for Software Estimation

Pattern Analogies are agnostic in terms of what overall software cost/effort estimation methodology is used. Pattern Analogies deals only with LOC estimation. You can plug that calculation into a number of mathematical models to figure out the effort in man hours. You could use a model as simple as multiplying the LOC times your team’s LOC per hour average to get the number of hours the project will take. On the other hand you could plug the LOC estimate into a complex mathematical model like COCOMO II or the regression-based model from Humphrey’s Personal Software Process. The choice is up to you and your team.

The complexity of the model that you should use to calculate effort, duration, etc depends on the level of accuracy you need in your prediction. The more factors the model takes into account, the better your prediction will be, and the more factors the model takes into account, the more involved the process of using the model becomes. Again, no matter what model you use to estimate effort, Pattern Analogies can give you the LOC estimate to feed it.

Case Study

Suppose we need to develop a currency converter in Java. To do this we will use a JFrame component that contains a JList element on the left and two text areas on the right. (I will not go into details about layout management for this application). Every time the user selects a new currency type on the left list, the program takes the dollar value on the right and transforms it to the currency type specified in the list and then outputs the value in the second text field. When the value on the top text area changes the program updates the value on the second text area. (See Figure 3 for a screen shot of this application.)


Figure 3: Currency converter application.

This application will need a JFrame to contain the components, a mediator for the interaction between the list and text fields, commands and command holders for the list, list elements, and the top text field. We will also need extensions of JTextField and JList that implement command and command holder. The list elements are going to have a CurrencyCalculator type member variable. We will need one currency calculator for each type of currency. Figure 4 shows a rough UML design of the application.


Figure 4: UML design diagram for currency converter.

Let’s divide the pattern instances of Figure 5 in two categories: the ones actually representing patterns and the ones not representing patterns. For the patterns we have a the mediator we discussed earlier, and a GUI component for which I counted 7 LOCs per widget in the GUI component. The estimates for the Custom Text Area and Custom List I got from looking at how many extra LOC the common list and text area will need to implement CommandHolder. For RecalculateConversion I got the average of all my previous command objects (I use commands only to call mediator methods, so their size is quite constant). For the components not representing patterns I used other techniques and assumptions, as described in the "Comments" column in Figure 5.

As we can see, the estimated size of the currency converter application will be 156 LOC. With the LOC estimate we can use, as an example, COCOMO II to calculate effort and schedule. COCOMO II is a mathematical model in which you define different variables that describe your development team. Variables can be very low, low, normal, high, or very high for your team.

A common variable is process maturity. So, for example if process maturity is high for your team you will have to use a particular coefficient in the final calculation for effort. The coefficients are calculated from a number of case studies used by the authors of COCOMO II. You can also calculate your own coefficients if you have enough projects. You don’t need to know how to do all these calculations. There is freely available software that does all this calculation for you.

Estimation with Pattern Analogies for Agile Teams

Now that we have technique for estimating software size, we have to decide when to do this estimation. Remember, the more you know about a project the better your estimation is going to be. An agile team will usually not work in a waterfall lifecycle, so I will not talk about doing estimation in that lifecycle. I will talk about size estimation in the spiral lifecycle and in the Extreme Programming lifecycle.

If you are using a spiral lifecycle you should first do an overall estimate before starting the project with a rough architecture. This will help you plan the cycles. In the planning stage of each cycle, create a new estimate for the entire project. At this stage you will have better knowledge of the overall system and your overall estimate will improve. With this information you should already be able to see if the next phase will completed or not be completed on time and can make scope decisions accordingly. After the design phase of each cycle, redo the estimate for the part of the system that you are working on in that phase. This will allow you to have a clear view of how long the phase will take. You need this updated information to make better design tradeoffs.

If you are in an Extreme Programming environment you should do a rough architecture to the granularity of patterns, after you have all the user stories. This will be the base for your first estimate. Make sure that you build your total estimate by doing an estimation of each user story. Update the estimate and the design after you finish each user story. Redo the whole estimate with the new information.

The reason why you calculate and recalculate so often is because the more complete the system, the better your estimate will be. So if you have to do staffing and scheduling decision in the middle of the project, you are going to have much better data on which to base these decisions. Also, this way you can give upper management and your team a clear view of where the project is and how fast the project is moving.

Keys for Success

The use of the Pattern Analogies technique requires four things from the development team:

* That everybody on the team uses design patterns as building blocks as much as possible and wherever it makes sense.
* A common vernacular and implementation style for patterns employed throughout the team.
* An overall system design composed from pattern-based modules and components.
* A library of "rules" and statistics for estimating the size of pattern instances. This is built up over time, accumulating estimation accuracy for successive projects.

Conclusion

Using the Pattern Analogies technique will not only provide a way to estimate size, but will also result in a better engineered product (stemming from of the careful use of patterns). If a developer understands design patterns, he will easily understand Pattern Analogies. Just remember, the job of an oracle was dangerous in antiquity and is still dangerous today. So make sure everybody in your organization understands that you are providing them with an estimate—not the exact future.

Automating Software Development Processes

Automating repetitive procedures can provide real value to software development projects. In this article, we will explore the value of and barriers to automation and provide some guidance for automating aspects of the development process.

Although few experienced developers and project managers would argue the merits of automating development and testing procedures, when push comes to shove many teams place a low priority on implementing automated processes. The result is usually that, if automation is considered at all, it is given lip service early in the project life cycle, but falls quickly by the wayside.

Experience teaches us over and over again that trying to run a "simple" project by implementing a series of "simple" manual protocols, backed by "simple" written (sometimes, even just verbal) instructions, just doesn’t work well. Even so, many of us still tend to allow ourselves to start the next project with the thought that the manual, protocol-based method will "do just fine."

After all, aren’t we all professionals? Can’t we read a set of simple instructions and just be disciplined enough to follow those instructions when the time is right? Isn’t it a waste of time and money to invest in automating procedures for such a small project? The development team is only a half-dozen people after all—and the argument against automation goes on and on.

If you’re a developer or tester who enjoys spending your time actually adding value to your project, rather than repeating the same routine tasks over and over, you’ll want to consider advocating the concept of automation to your team (especially, to your project manager). If you’re a project manager who’s committed to maximizing the talents and time of the members of your technical team, as well as minimizing the risk of your project failing to deliver on time and on quality, you will want to encourage your team to invest the necessary time and effort required to automate the types of tasks that will be identified in this article.

Why Should I Automate?

You may already be familiar with many of the benefits of automating development processes. Some of the more commonly cited ones are:

Repeatability. Scripts can be repeated, and, unless your computer is having a particularly bad day, you can be reasonably certain that the same instructions will be executed in the same order each time the same script is run.

Reliability. Scripts reduce chances for human error.

Efficiency. Automated tasks will often be faster than the same task performed manually. (Some people might question whether gains in efficiency are typical, noting that they have worked on projects where, in their view, trying to automate tasks actually cost the project more time than it saved. Depending on the situation, this may be a real concern. In addition, automation might have been implemented poorly or carried too far on some projects—but keep reading for more on what to automate, and when.)

Testing. Scripted processes undergo testing throughout the development cycle, in much the same way the system code does. This greatly improves chances for successful process execution as the project progresses. Automated scripts eventually represent a mature, proven set of repeatable processes.

Versioning. Scripts are artifacts that can be placed under version control. With manual processes, the only artifacts that can be versioned and tracked are procedure documents. Versioning of human beings—the other factor in the manual process equation—is unfortunately not supported by your typical source control system.

Leverage. Another big benefit to automating is that developers and testers can focus on the areas where they add real value to a project—developing and testing new code and features—instead of worrying about the underlying development infrastructure issues.

For example, instead of requiring everyone to become intimately familiar with all the little nuances of the build procedure, you can have one person focus on automating the build and have that person provide the team with a greatly simplified method of building, hopefully as simple as running a command or two. Less time spent on builds leaves more time for the tasks that add the most value to the project.

What Should I Automate?

If you’re convinced that automating your development processes is a good idea, the next logical question is which processes should be automated. While the answer, to some extent, is different for every project, there are some obvious ones, as well as some general guidelines that I’d like to offer. Some of the typical targets for automation are:

* Build and deployment of the system under design.
* Unit test execution and report generation.
* Code coverage report generation.
* Functional test execution and report generation.
* Load test execution and report generation.
* Code quality metrics report generation.
* Coding conventions report generation.

The above list is obviously not exhaustive, and every project has its own unique characteristics. Here’s a general, and perhaps obvious, rule of thumb to help identify any process that should be considered for automation: Consider automating processes that you expect to be repeated frequently throughout a system’s life cycle. The more often the procedures will be repeated, the higher the value of automating them.

Once a process has been identified, spend a little time investigating how you might be able to automate the process, including researching tools that could assist with automation, and estimating the level of effort required to implement the automation versus the total cost and risk of requiring team members to manually perform the procedures. As with any other business decision, it really should come down to a cost versus benefit analysis.

You probably noticed that the term "report generation" appears in the above list of automation candidates. The repetition points out another important aspect of automating development processes: the end result of every automated process execution should be a report that is easily interpreted by the team. Ideally, such reports will focus on making anomalous conditions (for example, test failures) obvious at a glance. Also, these reports should be readily accessible to the appropriate team members.

In many situations, it’s even a good idea to "push" reports to the team (perhaps via email or RSS), instead of requiring people to remember to go out and search for them. The basic idea is that the development team should be notified as soon as possible when problems are introduced to the system or environment. A side benefit is that management is provided a more tangible real-time view into the progress and health of the project than is possible with team status reports alone.
When Should I Automate?

Consider automation as soon as you realize that team members are starting to execute a process that meets the criteria discussed in the previous section. For example, automating the build process when the project is nearly over provides very little benefit. It can, however, save dozens, if not hundreds of hours when automated as soon as development begins.

However, before you go off and start automating everything, one caution: be reasonably certain that the procedure will be required for your project, that it will be repeated more than two or three times during the project’s life cycle, and that you understand the steps that need to be executed for the procedure. There are some things you just don’t do very often, and in these cases the costs may outweigh the benefits.

Even for processes where automation is warranted, if you’re truly guessing at the steps involved, there’s a high likelihood that you’ll end up re-writing the whole thing. Re-writing is much different than fine-tuning or refactoring the automated scripts. Refactoring is expected, but scrapping and starting over again means you tried to automate before you understood the process you were trying to automate.

Another danger is allowing your project to become the "proving ground" for automation for the entire organization. If your organization doesn’t already have the infrastructure in place to support development automation, you should provide what makes sense for your project, but try not to allow your team to lose sight of your project’s goals. A project team whose goal is to deliver working software is in no position to develop enterprise-wide strategies and tools. Trying to do so is asking for trouble.

If outside entities begin to get involved in these efforts, I’d suggest that you recommend that a separate project be undertaken to work out the automation infrastructure for the enterprise. Members of this "automation project" team are free to review your project’s implementation for ideas or for use as a launching point. Once this infrastructure is in place, the question of When to automate? is easier to answer for future projects, and less costly.

Obstacles to Automation

If there are so many benefits to automation, why don’t we see more of it on software projects? All software development teams struggle to balance the need to show immediate results with the long-term goals of the project. There are many obstacles to implementation of development process automation. Here are a few of the more common ones:

* Stakeholder pressure. Faced with the desire to show early and rapid progress, teams often overlook or choose to ignore practices that do not seem, at first glance, to directly contribute to getting working code up and running as quickly as possible.
* Focus on core requirements. The team has an overwhelming desire to begin producing working code and tests as soon as possible. Writing code that can be demonstrated is much more satisfying to developers than writing automation scripts.
* Lack of management support. Management may not have a good understanding of automation—how it works and/or the costs and benefits.
* Lack of organizational support. There is no enterprise-wide policy or infrastructure for development automation (standards, tools, expertise, etc.)
* Lack of follow through. Even with the best of intentions, teams can quickly lose their commitment to plans for implementing automated processes.

As with any fundamental change, there must be someone who’s both committed to the concept and who has the authority to make sure that teams follow through with the plan. Not following through is much worse than never having taken the time to investigate and discuss automation. First of all, there are probably other problems that you could have successfully solved using the time and resources and, second, if you fail to implement any automation, when the idea surfaces again people in the organization will point out that it didn’t work the "last time we tried it".

Selling Automation

As implied in the preceding section, the primary obstacle to automation is short-term thinking. Therefore, your primary task is to get your team and management to pause to think about overall project costs and schedule, instead of focusing solely on meeting next week’s promised deliverables. The message should be clear. Meeting milestones is critical, but it’s very possible that you could meet that next milestone to the detriment of the project. But before you try to convince people who are under a great deal of pressure that it sometimes makes sense to slow down momentarily in order to improve the chances for success, you’d better have more than a set of maxims at hand.

This is where Return On Investment (ROI) can help. Most people in the business world, regardless of field, have at least a basic understanding of ROI and agree that, when based on valid assumptions, it can be one of the best mechanisms for evaluating whether or not to take action or adopt a particular solution. The process of calculating ROI is well beyond the scope of this series of articles. However, the Suggested Reading section at the end of the article provides a link to an outstanding article from Edward Adams that describes how to do this, as well as some additional advice on selling the case for test automation, in a very straightforward and easy to understand way.

Although many managers really like to see quantifiable benefits, such as ROI, before making major decisions about adding tasks to the project schedule, others have sufficient experience with software development and are just as comfortable with a common-sense explanation of automation’s benefits. With that in mind, rather than going into a full-blown ROI calculation, let’s just take a look at one scenario that most projects will face at some point to see, from a common sense perspective, the likely outcome of a purely manual process versus a process that had been automated early on in the project.

Manual testing approach. Your team is in the final month of a twelve-month project, and the system test team is starting to perform regression testing by manually re-running the entire set of test cases. Some test cases haven’t been executed in several months, because, frankly, the testers have just been spread too thin, and, due to perceived pressure to deliver, management chose to move testers onto other tasks as soon as they were finished wrapping up a set of test cases.

Regular regression testing was put into the schedule early on, but repeating the same tests over and over throughout the project was (whether or not anyone wants to admit it) eventually considered a "nice to have" and definitely much less critical than other, more pressing tasks. Now some of the regression tests are failing and the test team is logging defect reports.

The development team decides to manually re-run the entire unit test suite (which also hasn’t been executed in a few months, because it wasn’t automated) to hopefully zero in on the source of the problem. The developers find two problems as a result of the unit test run: first, apparently there were never any unit tests written for the parts of the system in question; and second, there are other, seemingly non-related unit tests that are failing. To make things worse, the components that are failing were developed by someone who’s no longer with the team. Even if he were available, he’d probably require significant spin-up time to re-acquaint himself with the failing code, since he hasn’t even looked at it for months.

Automated testing approach: Early on in the project, the decision was made to automate at least the following: unit testing, code coverage, and system testing. The same bug that caused the situation above was introduced in month three of the project. The code coverage report that’s automatically generated as part of the daily build clearly indicated to the development team that there were no unit tests for the related components as soon as those components were placed under source control. Therefore, the developer went back and immediately implemented the unit tests.

After implementing the unit tests and working out the bugs that were initially revealed as a result, the developer checks the unit tests into source control. The daily build automatically picks up the new unit tests, the coverage report reflects that the components are now being tested (as well as to what extent they’re being tested), and the unit test report indicates test success or failure each day. As soon as any change is made to the system and placed under source control, the team will become aware of the impact that change has on the overall system, while the changes are still fresh in the minds of the implementer.

The system tester who’s responsible for testing that same set of features now designs his test case, implements the tests, reports any defects he discovers, etc. until he feels that the code and test case are operating as expected. He then uses the test automation tools to record the execution steps for the test case and checks the resulting scripts and files into source control. The next day, the automated test runner picks up his new test case for execution and the results become available to the test team as part of the daily test run. The development and test teams are leveraging the automated tools and processes for proactive detection as problems are introduced.

Without performing a detailed ROI analysis, which of the two above scenarios is intuitively more desirable? Assuming that the exact same bug is introduced into the system at the same time—month three of the project—in each case, which approach do you think will result in the least expense and least risk? Which will result in the least stress to the team and to management?

Start Small

Unless you’re lucky enough to have a team with multiple members who’ve had a good deal of experience automating development processes, you won’t want to try to take the idea of automation to the extreme. Trying to do so with an inexperienced team or a team that’s not completely sold on the idea of automation will likely result in failure, as both expectations and barriers are set high from the start. Pick one or two areas where you believe there is both high potential value for automation and a low risk that the team will fail implementing automation. You might ask yourself the following questions:

* Which processes will likely be exercised the most frequently?
* Does someone on the team have the experience and/or skill set to implement the automated process?

It’s better to succeed in a small way than to fail in a big way. Your successes just might build on themselves, and when you suggest additional areas for automation, either later in the project or on your next project, you may face little or no resistance at all. To reach the ideal takes a long-term plan, management commitment, adequate tools and training, and of course time. If it were easy, everyone would already be doing it.

Best Practices for Object/Relational Mapping and Persistence APIs

Over the last decade there has been a lot of effort put into object/relational mapping, which refers to techniques for resolving the mismatches between the object-oriented world, with its encapsulation of data and behavior, and the relational world, with its tables and columns. There’s not only a difference in terms of data types (and complexity of these data types) but also in terms of relationship types. The object world has a variety of relationships (aggregation, composition, association, inheritance) which cannot be mapped directly to the database world.

The general topic of object/relational (O/R) mapping can be divided into two areas of concern: the mapping itself and the persistence API. The persistence API not only acts as an indirection layer for the database, but also hides the mechanics of mapping the objects to the database tables. A good persistence API should do this while not constraining the object modeling in terms of data types and relationships.

In this article I will begin with a discussion of home-grown vs. off-the-shelf persistence solutions, including areas to consider when deciding between the two, and advice for choosing the best off-the-shelf solution to meet your needs. I will also share suggestions and advice from my own experiences with O/R mapping and persistence APIs. It is not my intention to explain all of the background details of these topics, but to focus on "best practices."
Home-Grown Mapping

I have been directly involved with both home-grown (custom built) and commercial-off-the-shelf (COTS) mappers, and have also observed several other home-grown mapper implementations, each approaching the subject in a specific way. In my experience, the primary drawback of "rolling your own" persistence mapping implementation is that limited resources do not allow for enough time to think everything through, to improve the framework over time, and to backtrack if you realize that design changes are needed.

Because an O/R mapper is a generic piece of software, it is typically hard to explicitly list which aspects are of the most importance to you (and if you build your own, you will not be able to focus on all of them). I do not mean to say that you could not envision a good design, but that it would take a lot of time and effort to fully implement a solution that meets all of your needs.

I observed the following limitations in just one home-grown O/R mapper:

* It provided no support for association relationships; only containment relationships were supported. This was a serious constraint when defining the object model.
* It offered no support for transactions.
* It only supported one RDBMS.
* The API was not type safe, which caused a lot of errors that could only be detected at run time.
* Testing of the O/R mapper was underestimated. Not only were there a lot of possible paths through the code, there were also stability and performance issues to be considered.
* The designer had to create and maintain the object model by editing a plain text file without any special editor. I know the saying "a fool with a tool is still a fool," but not being able to visualize one’s own object model is like walking around in the dark—you don’t see where you are heading and you have difficulties in pointing the others in the correct direction. In this case, the object model, which was like a red ribbon winding through the whole architecture, could not be clearly communicated to the team.

I hope these examples are enough to help you to avoid stepping into a home-grown solution. Of course, a home grown O/R mapping project would be fun to do from a development point of view, but a COTS (commercial-off-the-shelf) O/R mapper will be cheaper—unless you consider O/R mapping as one of the core competencies of your business.

Selecting a COTS O/R Mapper

Below I list some criteria you might want to consider when selecting an O/R mapper.

* Consider whether the tool will restrict your modeling freedom too much. For example, many tools don’t support relationships on abstract classes. A workaround for this is to duplicate relationships on concrete classes, which is less ‘OO’, but works for these tools.
* Consider whether the O/R mapper that allows you to model visually (preferably using UML).
* If UML is important to you, ensure that you can either import UML into the O/R mapper, or that you can export UML from the O/R mapper.
* Take a close look at the programming model the O/R mapper imposes and see whether it is compatible with the things you want to get out of it.
* Look at the range of mapping possibilities to ensure that the kinds of relationships you envision between your objects and tables will be supported. Typically, most O/R mappers support a large range of features, but not every mapper supports every type of relationship.
* Assess the performance, even if you think you do not have a lot of performance demands. Testing the performance can also give you a chance to learn and assess the API.
* If you prefer or need to start with an existing database schema and then map objects onto it, assess whether the O/R mapper supports the database system you want to use.

Selecting a Persistence API

The O/R mapping features are only part of the story. The other part of the story is the selection of a good API for persisting objects, and this part has a lot more visibility to your development team than the O/R mapping part. While the O/R mapping functionality will only be exposed to a few team members who are dedicated to maintaining the persistence layer, the persistence API defines the interface that the whole development team will use.

Persistence APIs can be divided into two categories: transparent and non-transparent.

Transparent Persistence APIs

A transparent persistence API hides the persistence completely. A transparent API does not need to have a lot of methods; a load and a save method is sufficient most of the time. Typically, a lot is defined declaratively instead of procedurally. Hibernate and JDO are examples of transparent persistence APIs. Let me clarify with an example:

An Insurance object can contain 0-n Warranty objects. The client application updates an attribute of an Insurance object. Semantically a Warranty is contained by an Insurance, so when you update an Insurance it is possible that you implicitly update its Warranties. According to the requirements, this might be a correct design, but it could have a negative impact on performance, especially if I am not aware that the implicit update of the Warranty objects is happening. When I only have modified an attribute of the Insurance object, I should be able to limit the persistence manager to this functionality.

The beauty of this is the "magic" way in which the persistence manager knows what to do. The negative side is that the persistence manager is thinking in your place.
Non-Transparent Persistence APIs

A non-transparent persistence API has a lot less "magic" inside of it. When compared to a transparent persistence API, it has a rich API, offering a lot of control to the user of the API.

Consider a transparent persistence API with a single method Persist(). This persistence method does all the magic behind the scenes, like checking whether there are associations that potentially need to be persisted as well. Although this might sound attractive, when selecting a persistence API, ensure that you can optimize. When touching associations, it is possible that the associated objects haven’t been persisted yet. What should the Persist() method do? Persist those objects first? I say that it is better to put the client in control.

To illustrate the power of a non-transparent persistence API, I’ll use the SimpleORM API as an example. Consider an insurance class with two methods. The first method will explicitly load all of the children linked to this object:

insurance.getAllChildren(insurance.Warranties)

The second method will only list those items that were added to the object in memory and the ones that are already retrieved from the database:

insurance.getRetrievedChildren(insurance.Warranties)

The advantage here is that the user of the API can "see" what he is doing, and can make better decisions regarding performance costs. (Also during code reviews this visibility can make life a lot easier.) In contrast, due to the fact that a transparent persistence API has a very generic interface (e.g. Save() and Load()), it also creates the illusion that database actions are cheap.

The cost for the non-transparent API is that the interface is more complex than a transparent persistence API. However, if you are planning to write your own persistence API, I recommend a non-transparent persistence API.

It is not my intention to classify all transparent persistence APIs as "do not use." But if you are considering a transparent persistence API, I would advise you to assess its performance carefully.

In case you are not satisfied with the persistence API that is provided to you, you can also decide to wrap it such that it maps onto your needs. In the next section, I’ll elaborate on some good reasons to wrap a COTS persistence API.
Wrapping a COTS O/R Mapper’s Persistence API

After you have selected a COTS O/R mapper, you should also decide whether to use its persistence API directly or to wrap it. Good reasons for wrapping the O/R mapper’s persistence API are:

* You want to add some extra logic (for example, field validation is not part of the O/R mapper, but you want it to be an inherent part of the persistent objects).
* Some O/R mappers generate code for the persistent objects, others expose a generic API. Typically, the generated persistent objects are type safe, while the generic API is not. Type safety is a good reason for wrapping the O/R mapper’s persistence API.
* Apply the subsystem principle: you want to avoid a tight coupling with a specific O/R mapper. Therefore you treat it as a subsystem and work interface-based.
* You want to limit the features that your clients can use. For example, your mapper might support the use of direct SQL while you don’t want to expose such a feature.
* You might want to expose certain services in a different way. For example, you might want to introduce a query object rather than to expose an OQL query interface.

Of course you need to place everything in perspective: if you are creating a throw away application, you don’t need to worry about aspects such as maintainability, extensibility, and resilience. For strategic applications however (commercial software products, product families, core business applications, etc.) you really should spend some time evaluating the pros and cons of different approaches.

A bad reason to wrap a persistence API is that you think that you will be able to boost performance. In general, you won’t be able to do this because the performance is inherent to the internal design of the COTS component. Only in rare occasions you will be able to turn the performance to your advantage by wrapping the COTS component.

A better way to approach performance is to invest some time in seeking a usage pattern that is optimal for your situation.
Wrapping and Object Management

Another way to discriminate between persistence APIs is the way objects are managed.

In one technique, the persistence manager is an object factory (using the factory pattern):

MyPersistenceManager mgr = MyPersistenceManager().Instance;

Insurance insuranceObj = mgr.CreateInsurance();

mgr.Persist(insuranceObj);

An advantage of this approach is that the manager always knows the current state of the object (new, retrieved, already saved). The manager can use this information to its advantage, resulting in good performance.

In other APIs, such as JDO, the objects are not created within the context of the persistence manager. The persistence manager takes an object in and determines what to do with it. In this JDO example, the manager (mgr) does not know the current state of the insurance object (unless perhaps it caches the information):

PersistenceManager mgr =
persistencemanagerFactory.getPersistenceManager();

Insurance insurance = new Insurance();

insurance.SetPolicyNumber("2005001001-110");

mgr.makePersistent(insurance);

Okay, by now I hear you saying "All of this is very interesting, but why is it of my concern?" Two reasons:

First, consider object management when you want to wrap the persistence API. Performing the wrapping without knowledge of the pros and cons of different persistence API approaches would be very dangerous.

Second, consider the performance implications. In the case of the object factory mechanism, you can be quite sure of decent performance. In the latter mechanism, it’s advisable to assess the performance by means of some unit tests or architectural prototypes.

Disconnected Objects

A typical disconnected object scenario starts with the retrieval of the object from the database in the server app, then streaming it to the client app, then in-memory modification on the client app, and finally streaming it back and storing it to the database again in the server app. The way the persistence framework handles disconnected objects has quite a big impact on the performance.

One way to improve the performance of disconnected object persistence is through merging, as in this JBoss example:

Insurance insurance = Util.deserialize(input);

// Changes are monitored as of now.
entityManager.merge(insurance);

insurance.setSubscriptionDate(d);

entityManager.persist(insurance);

In the above example, only the change of the subscription date is stored in the DB.

Another area of performance concern is keeping track of changes vs. figuring out at the time of persistence what changes have occurred. The latter technique is more dangerous for performance, especially when complete object trees are being saved. The algorithm to find out the changes has to be fast, or the original data needs to be cached in order to achieve a good performance.
O/R Mapping Best Practices

I have distilled the following "best practices" from my experiences:

* Don’t work against the O/R mapper’s persistence design. Rather, take the O/R mapper’s design principles as a constraint and exploit them. If you don’t, you’ll have to pay in terms of efficiency, performance, etc. Also, make sure that you know the basic concepts of the O/R mapper’s design.
* Wrap the O/R persistence API and treat it as a subsystem, such that you work interface-based, which eases the prospect of switching later. I don’t say that it won’t hurt to switch to another O/R mapper, but at least the pain can be isolated.
* Check the querying capabilities of the O/R mapper’s persistence interface. Especially, check whether aggregate functions can be used and whether you can query for raw values rather than plain objects. Objects can be too much of a good thing (object bloat) when, for instance, you just need a couple of values to fill a grid.
* Implement field metadata wisely. Generate field metadata (such as size, etc.) in-line in metadata classes instead of using reflection. This improves performance, and also makes it easy to debug the code.
* Put field validation at the level of the metadata. Make sure to expose your field-level business rules such that you don’t require a round-trip from your client to your server application in order to know whether or not the object is in a correct state to persist it. Tools such as SimpleORM suffer from this (typically field validation is foreseen internally, but not exposed).
* Be careful when calling a field a "mandatory" field. Typically, O/R mappers foresee the ability to tag certain fields as mandatory. In my experience, I have seen that this construct is used too much. Typically fields are mandatory dependent on the state of the object and such interdependencies can seldom be expressed. Therefore, the best practice is to limit the mandatory fields only to those fields that make the object incorrect within the application domain.
* When defining the persistence API, guard its consistency and ease of use. When specifying the interface of your persistence API, make sure that the parameters of each persistence method are consistent with the rest of the interface. As an example, the following dummy interface is not consistent because in one case an enumeration is used (RelationshipName), and in the other case a string:

SaveRelation(RelationshipName name, object value)
SaveAttributeToObject(string attributeName, object o)

* I would suggest to either go for a type-safe approach or for a generic approach, but not to mix them in one interface. If you want both of them, then put the generic methods on a separate, more generic interface. Don’t make the interface more complex than it should be.

* Make sure that the API can be used in a way that a client application can take advantage of its knowledge to optimize performance.

What Is A Professional Programmer?

How do people become professional programmers? Many people go the "traditional" path through a computer science or software engineering education and from there into professional programming work.

Others become professional programmers by accident. A person writes a small program to help at work, and their workmates say, "Oh great, you can write programs! You're our programmer now!"

Other people start out as hobbyists and follow a less traditional path, not always getting a degree, but clearly wanting to be programmers from the start and working actively towards that goal.

I've been a hobbyist programmer since I was 6. I wasn't writing anything amazing back then but I had started writing and soon found it was absorbing most of my time. Since I never really stopped, that gives me 24 years "programming experience" and counting.

At first I was into writing computer games. Later people asked me to write programs for them, and sometimes I even got paid. From this I learned that software is always for something. Programs are not self contained worlds of their own. People expect things out of a program that have more to do with Japanese or Geophysics or Engineering (or whatever they've got in mind) than with how a computer works. I had to learn something about all those domains in order to write programs for them.

At university it didn't take long before I was a tutor, and that's where I found I enjoy teaching, and especially enjoy teaching programming.

While I was at university I got my first "real" job, writing Visual C++ code for a financial database company. In terms of design and theory it was lightweight stuff. But in terms of working with others on a large project I was being thrown in the deep end! They had gigabytes of source code, growing cancerously through the efforts of a dozen developers of wildly differing skill levels.

In spite of my programming skills being well above average there, I learned to settle for being a junior programmer, a little fish in a large pond.

Skipping along a few more jobs and a lot more years, today I am a senior developer in a small research group—a big fish in a little pond. I've had to teach my co-workers a lot about professional programming, because most of them haven't been in industry to get that taste of what large code bases and diverse skill levels do to programs if you aren't using those "professional" skills to keep everyone pointed in the same direction.

There's quite a gap between "being able to program" and being a "professional programmer." It took me 15 years to go from beginner to hotshot programmer, then another 10 years to go from hotshot to professional—and I'm still learning.

Whatever the path we follow, most professional programmers have in common the fact that they learned to code first and how to be a professional later.
The Meaning of "Professional"

So what does it mean to be a professional programmer? What does it mean to be a professional anything? Some definitions simply say to be a professional is "to make money from a skill," but true professionals also have a set of qualities often described as "professionalism." In my opinion, these qualities are: trustworthiness, teamwork, leadership, communication, constant updating of skills, an interest in minimizing risks and accountability. Each of these effect the professional programmer in certain ways.

Trustworthiness The concept of trustworthiness applies in several different ways for programmers. Can you be trusted with a job? To perform a task without someone checking up on you? Can you be trusted to ask for help when you need it?

If you're given clients' data or have signed a non-disclosure agreement, then you are being trusted to respect privacy. You are trusted to check license agreements on third party tools or libraries and to get licenses or permission as required. And like any professional you are trusted to simply do a good job.

Teamwork Will you genuinely cooperate with your team mates? Will you work to mutual advantage and not just your own? Can you trust your team to work with you? Can you do your share of the work and trust your team to do the rest? And can you accept your management (and sometimes even clients) as part of the team, everyone trying to get the same job done?

Leadership Showing leadership means both earning respect from others and knowing what to do with it. Recognize the skills of your team members, and make sure you can offer each person challenges and development without exceeding what they can cope with at a given time.

Leadership involves not always getting to do the "fun" parts of a project yourself (that scary "delegation" word). It also involves not asking anyone to do a task that you wouldn't be willing to do yourself. It's not just the managers and lead programmers who need to show leadership, it's any professional programmer. The best programmers to work with are the ones that know what's going on, not just their little tasks.

Communication Respecting the people you work with, and your clients, enough to really listen to them is a critical part of communication. Teamwork can't happen without good communication, nor can accountability.

Communication is critical for helping clients to produce usable specifications and feedback. Will you question whether the specs you are given really will serve the purpose that the client has in mind?

Communication skills help with making meetings timely and effective. A professional's communication is effective and to the point, whether in person, in email, on the phone or in written documents.

Documentation at first seems like a programmer-specific concern until you consider how many people require documentation in a serious project: other programmers need high level, API level and in-code documentation; managers need planning, progress, and bug documentation; lawyers need proof of what was done and when; and users need documentation on how to use the software.

Updating Skills Keeping your skills up to date involves staying aware of what's going on in your industry. What are the current ideas about methodologies like eXtreme Programming? What libraries and tools are out there that might support your project? What are the current refactoring tools? How about standards, file formats and protocols? Are you up to date with Unicode, XML, SQL, and all the other acronyms? Perhaps you're missing out on something if you're not. What platforms are your potential clients using? Should you be learning about cross platform development?

Basically you need to possess a genuine interest in your field, and to read broadly so you know what's out there and which areas to then read deeply about. You also need to accept that even (or should I say "especially") the very best programmers are still learning.

Minimizing Risks Familiarity with best practices, combined with a healthy dose of common sense, will take you a long way towards managing risks. Professional programmers keep track of known bugs or any other change they intend to make. Bugs are risks, and a simple database can prevent you having a product ship with bugs you'd simply forgotten.

Another risk that's often not properly considered is any and all changes to the source code. Source is your livelihood and any change can be a mistake. There's good software out there that will keep track of every revision of your source code and even help merge code that multiple people have changed.

Professional programmers are careful to do enough testing. A software company will generally have testers but the developers need to know how to get the most out of testers and also how to write their own unit and regression tests to make sure every change in behavior is noticed and checked by a human.

Keeping your code simple and well styled is another commonly overlooked way to manage risks. If anyone can look at the code and see right away what it does, you are far less likely to find bugs in it later, and you are less likely to have a junior programmer attempt to change something without understanding it first.

Another risk is the client changing their mind, or more often changing their specifications because they've realized it wasn't what they had in mind. Write your code to be modular and reusable and you won't have any trouble adapting it to changing needs.

Accountability Writing code for others is a responsibility. You need to make sure your software is reliable. You need to make sure you and the client truly understand the requirements and specifications. You need to have documentation of your work, all current and past bugs, your progress, any problems, signed-off milestones, and more. You are also required to know about some basic legal issues, like software licensing, the terms of your employment contract, and intellectual property law.

* * *

As you can see, there is a huge gap between "coding" and "professional programming." Most programming courses focus on the coding side of things, and the professional skills tend to be glossed over or not covered at all. I have found myself regularly teaching these skills to new co-workers, which highlighted the need for "professionalism skills training." Teaching my co-workers reminded me how much I enjoy teaching. I decided to teach more people by trying my hand at professional writing for a change.

Career Paths for Programmers

Recently interviewed for a Business Analyst position with the CIO of a large multi-national software development firm. This man was in charge of the company's worldwide IT operations, including offshore development projects, for which he was searching for qualified Business Analysts. The interview quickly became a casual conversation about current trends within the IT service sector, how the company was planning to take advantage of those trends, and, most importantly, how I could fit into those plans. It was during his evaluation of my skills that I asked how I fit and whether it was technical or business skills that were most valuable to his projects. The CIO summed up his advice about my career path with one small sentence: "Stay on the business side."

Business skills, in this CIO's view, were most important to his future projects and the industry as a whole. His reasoning was that he could train anyone in the technical skills he needed for a project, but finding those people with the necessary business skills to guide an IT project to success was something that could not easily be obtained. He went on to say that he found it difficult to find people who could communicate on even the most basic of levels. I asked if my background as a developer would help in getting a business analyst job, and he conceded that although it's not a requirement, it certainly would help matters as long as I could prove that I wasn't "too technical."

His comments are consistent with the trend that all US-based programmers have observed since the late 1990's: global salary competition amongst programmers, and a growing view in big business of programming as a commodity skill. It's hard to compete with a developer in Russia or India who can work for a fraction of what I make minus benefits. The CIO managed to reaffirm the subtle, but major, shift from technical skills to business-technical skills in today's labor market. I gave weight to his viewpoint since the people in his position are the trendsetters of the technology industry. They are the ones who set the directives for a company's IT needs, and often, the requirements desired for the higher-paying positions.

I did a little research and found that the US Bureau of Labor Statistics Occupational Outlook Handbook predicts that computer systems analysts are expected to be among the fastest growing occupations through 2012. The Handbook describes a systems analyst as someone who may plan and develop new computer systems or devise ways to apply existing systems' resources to additional operations. It describes a computer programmer as someone who writes programs according to the specifications determined by systems analysts. (The book does not separately list business analyst as an occupation.)

According to the Handbook, in the US systems analysts held an astounding 487,000 positions in 2004 (up from 468,000 positions in 2002) compared with 455,000 jobs in 2004 for computer programmers (down from 499,000 in 2002). The Handbook also states that employment for computer programmers is "expected to grow much more slowly than that for other computer specialists." And recent estimates by the Economic Policy Institute have put the number of jobs being offshored at approximately 330,000 to 500,000 jobs. About 100,000 of those were full-time computer programming jobs.

The key to maintaining a good employment outlook in IT, it seems, is to move out of programming and up into more business-oriented IT positions such as systems analyst, business analyst, project manager, or systems architect. However, a computer programmer can't just decide to become a systems analyst or project manager overnight. The journey takes time and requires the right amount of experience and learning to be successful.

Making the Shift

So you've seen the statistics and watched as the jobs in your market slowly disappear. You want to move more to the "business side," but you don't quite know how to do it. As I'll describe next, making the shift can be done on-the-job by gaining more responsibility, polishing up your problem-solving skills, and using creativity in your work.

I began my journey into systems analysis and design by accepting more responsibilities throughout the project I was on when things proved too overwhelming for my superiors. I gradually accepted more of the project management and business analysis responsibilities when the opportunity presented itself. For example, I would walk to Suzy in accounting and work out a new enhancement with her one-on-one rather than wait for my manager to do so. Over time, as my manager's confidence in my abilities grew, these responsibilities became a part of my job. It wasn't long before I became the Programmer Analyst, and ultimately the Project Manager, as new positions were created to fulfill demand for our work.

When the need arises, I recommend walking to the end user yourself and working with her one-on-one. Your manager will be relieved when he discovers that you are capable of communicating with his end-users, identifying their issues, and resolving those issues before they are brought up in the weekly manager's meeting. Even the best IT managers need a subordinate who is visible to the users who they can trust to get the job done. If a manager is slowly factoring himself away from the day-to-day workings of the project, welcome it. The higher visibility that you are obtaining can be translated into higher value—and that can result in a promotion. Over time, your increased interactions with more business-oriented people will make you more sensitive to business concerns.

A good subordinate has to be open-minded and creative. When solving problems, one has to always believe that there is a way to accomplish something, even if it's never been done before. Sometimes, just listening to the user will produce an idea. A lot of issues may come down to the business process that the system is attempting to replicate. I have had users actually solve a business problem for me just by listening to what they had to say!

Whether you're open-minded and creative or not, you can still work towards more business-oriented positions. After all, business systems analysts and project managers are only a small subset of the many positions opening up each year to address the issues of complexity through simplicity. If you love programming, you don't have to necessarily give it up.

Jobs To Pursue

Senior Technical Positions

Developers will often find that they may have to work side-by-side with the users to iron out difficult bugs. It can be difficult, if not impossible, to fix these problems when both parties can't communicate effectively. There was always a time in most of my work situations when the developer had to talk with the users or other developers directly to fix difficult issues. This is the programmer's chance to show management that he or she is someone who can communicate and utilize analysis methodologies—otherwise known as a "programmer analyst." A programmer analyst is also usually someone who has some years of technical experience, and a certain depth of technical knowledge.

Programmers who seek advanced technical skills without too much end-user interaction may find themselves gravitating toward the design & architecture side of the business. Although these types of positions are still relatively technical, they often involve making key decisions to address how the new system will fit into the organization's overall IT plans. In order to be successful, the architect needs to understand and control the elements associated with the utility, cost, and risk factors of the proposed solution.

System architects must make very educated decisions about how to decompose and isolate the different components that will be required, how to fit these components into the existing infrastructure, and in what order to implement each component. It can be a disaster to implement an online ordering system that isn't compatible with the organization's current accounting packages. The architect must identify these types of issues and present them to non-technical management in words they can understand.

Business and Systems Analysts

My job searches have suggested that business and systems analysts with a good programming background and a high-level of "business savvy" are becoming the next hot ticket. More and more organizations are finally hiring business analysts to explore, record, and recommend systems that fit the business—as opposed to the other way around.

The business analyst must often work with project managers, systems architects, and systems analysts, all of which are growing occupations that can make the difference between success and failure. In some cases the business analyst's responsibilities are being combined with that of the systems analyst or the project manager under the guise of "business analyst" or "business systems analyst." A quick search on Dice.com will reveal that many business analyst jobs have hidden deep within their job descriptions requirements to develop technical specifications or to guide and manage projects. My first business analyst job required both project management and systems analyst skills. These positions are sure to become more common as organizations struggle to reduce project failure and development time.

Project Management

According to the Bureau of Labor Statistics' Occupational Handbook, employers prefer project managers who possess advanced technical skills that have been acquired through work experience. The project manager is often responsible for hiring the staff, setting the schedule, and keeping track of the progress through every phase of development. This person is also responsible for assigning the work, dealing with everyday problems affecting that work, and making sure each analyst or programmer is carrying his own weight. The project manager can best carry out this function if he truly understands the work he is managing.

The project manager must also be a "people person" as well as a "technical person" in order to succeed. This individual must work with technical and non-technical staff at every level of the organization in order to succeed in his goals. Additionally, the project manager has to manage his team effectively to produce the desired product on time.

Management

The ultimate assignment for many IT professionals looking to move up the IT food chain is to become the manager. The Occupational Handbook explains that "employment of computer and information systems managers is expected to grow faster than the average for all occupations through the year 2014." These job opportunities are best suited for applicants with computer-related work experience and often require an advanced degree, such as an MBA. And of course, strong communication skills are a requirement for any management job in IT.

Skills To Develop

Okay, so you've heard all about what's required and where IT is going, but how can you capitalize on this new information?

My interview with the CIO and my experience in the field have shown me that companies want IT professionals who can understand what their business is and how to apply technology to make it better. Being able to follow directions is important, but being able to take some initiative and make your own judgments without handholding is equally important. The solution is to differentiate yourself from the traditional developer.

We have already discussed two ways of building up your current skills—acquiring business knowledge and advanced technical knowledge—but two other areas are important as well: communication and leadership.

Whether that CIO I interviewed with believed that communication skills could be learned or not is irrelevant. Everyone can learn to be a better communicator with practice. The difference is that communication skills take much longer to develop. Communication takes the right mix of experience and training to become effective. I have worked on this since my college days and have had great success in my career as a result.

I learned to communicate more effectively by dealing with those who couldn't. Many software users can't understand the technical side enough to describe any of their requirements in any type of detail regardless of their background. On the other hand, many technical people don't understand the intricacies of the business processes they are implementing because they can't openly communicate with the users. Learning to communicate, and having the patience to gain knowledge from the user, is an essential skill that many of my former and current coworkers don't have.

To add to your problem solving skills, instead of asking your superior or a more experienced programmer to help with a problem, take it upon yourself to find the answer to that complex problem. Before too long, you can be the one who others consult when there is a problem to fix or a new project to complete. Gaining problem-solving experience not only improves communication, it also improves your chances of moving into analyst and management positions. Eventually, you can do as I did and get your own project to manage.

The key to moving up the ladder at any company is to let them know what you know. Answer those questions, solve those problems, accept those new projects, and don't be too shy to share a better solution. It could mean the difference between being "just another programmer" or being the top candidate for a promotion.

The Human Impact of Software

My very first job in the computer software business was as an entry level help desk technician. I had been a computer user for many years (since Dad brought home the family’s first Tandy Color Computer), but truly I knew very little about how computers worked. Sure, in high school and college I had written a few BASIC and Pascal programs, but none of that knowledge had stuck with me. At that moment, I was vastly under qualified to support my new employer’s vertical market accounting software.

I joined this tiny software firm on the cusp of the 1.0 release of their first application. If I remember correctly, when I came on board they were in the process of running the floppy disk duplicator day and night, printing out address labels, and packaging up the user documentation. As I pitched in to help get this release out the door, little did I know that I was about to learn a lesson about software development that I will never forget.

The shipments all went out (about a thousand of them I think, all advance orders), and we braced ourselves for the phone to start ringing. In the meantime, I was poring over Peter Norton’s MS-DOS 5.0 book, which was to become my best friend in the coming months. We knew the software had hit the streets when the phone started ringing off the hook. It was insane. The phone would not stop ringing. Long story short, the release was a disaster.

Many people could not even get it installed, and those people who could were probably less happy than the ones who could not. The software was riddled with bugs. Financial calculations were wrong; line items from one order would mysteriously end up on another order; orders would disappear altogether; the reports would not print; indexes were corrupted; the menus were out of whack; cryptic error messages were popping up everywhere; complete crashes were commonplace; tons of people did not have enough memory to even run the application. It was brutal. Welcome, Dan, to the exciting world of software.

Eventually, we just turned off the phones and let everyone go to voice mail. The mailbox would fill up completely about once an hour, and we would just keep emptying it. We could not answer the phones fast enough, and when we did, people were just screaming and ranting. One guy was so mad that several nights in a row he faxed us page after page after page of solid blackness, killing all the paper and ink in our fax machine.

It took us months to dig us out of this hole. We put out several maintenance releases, all free of charge to our customers. We worked through the night many times, and I slept on the floor of the office more than once. Really the only thing that saved us was our own tenacity and the fact that our customers did not have any other place to go. Our software was somewhat unique.

It was obvious to everyone in our company what caused this disaster: bad code. The company had hired a contract developer to write the software from scratch, and, with some help from a couple of his colleagues, this guy wrote some of the worst code I have ever seen. (Thom, if by some slim chance you’re reading this, I’m sorry man, but it was bad). It was total spaghetti. As I learned over the years about cohesion, coupling, commenting, naming, layout, clarity, and the rest, it was always immediately apparent to me why these practices would be beneficial. Wading through that code had prepared me to receive this knowledge openly.

I stayed with the company for three years, and we eventually turned the product into something I am still proud of. It was never easy, though. I swear I packed ten years of experience into those three years. My time working with that software, that company, and the people there who mentored me have shaped all of my software development philosophies, standards, and practices ever since.

When I got some distance from the situation, I was able to articulate to myself and others the biggest lesson I learned there: software can have a huge impact on the lives of real people. Software is not just an abstraction that exists in isolation. When I write code, it’s not just about me, the code, the operating system, and the database. The impact of what I do when I develop software reaches far beyond those things and into people’s lives. Based on my decisions, standards, and commitment to quality (or lack of it), I can have a positive impact or a negative one. Here is a list of all of the people who were effected negatively by that one man’s bad code:

* Hundreds of customers, whose businesses were depending on our software to work, and who went through hell because of it.
* The families of those customers, who were deprived of fathers and mothers that had to stay up all night re-entering corrupted data and simply trying to get our software to work at all. (I know, because I was on the phone with them at three in the morning.)
* The employees of these customers who had to go through the same horrible mess.
* The owner of our company (who was not involved in the day-to-day operations), whose reputation and standing was seriously damaged by this disaster, and whose bank account was steadily depleted in the aftermath.
* The prominent business leaders in the vertical market who had blindly endorsed and recommended our software—their reputations were likewise damaged.
* All of the employees of our company, for obvious reasons.
* All of our families, significant others, etc.—again for obvious reasons.
* All of the future employees of the company, who always had to explain and deal with the legacy of that bad code and that disastrous first release.
* The programmer himself, who had to suffer our wrath, and who had to stay up all night for many, many nights trying to fix up his code.
* The family of that programmer (he had several children) who hardly saw him for several weeks.
* The other developers (including myself) who had to maintain and build on that code in the years to follow.

That’s a lot of people, numbering in the thousands--but only one developer’s code.

Traditionally, a large portion of the industry has relied on workstations from SGI, and OpenEye has supported a wide variety of platforms on which their customers deploy, including Windows, and multiple flavors of Linux and UNIX.


But recently, OpenEye has begun to include Mac OS X as a solution in their multi-platform development environment, and sees Apple as an increasingly attractive solution for their end users.

The decision to deploy Apple solutions is becoming more and more compelling in the pharmaceutical industry. Mac OS X with its UNIX base offers a growing, stable platform for users and developers to conduct their research. And Macintosh computers are an increasingly desirable option with the Intel Core Duo processors and top-notch 3D graphics support.

A Look at Multi-Platform Development in Mac OS X

While OpenEye has a strong product line, filling each niche of large-scale molecular modeling, they don't sell shrink-wrapped products. They offer highly specialized tools that are often developed in concert with customers who have very specific needs. The results have been outstanding, so as the product line grows, more and more customers are coming to OpenEye either simply to buy “finished” products or else to purchase toolkits to write their own in-house software.

A rapid development cycle and multi-platform toolkits dictate the need for a development environment that is efficient and portable. Compile times have become a key component to the efficiency of OpenEye's developers, so they were thrilled when they saw huge performance gains on Intel-based Macs.

Bob Tolbert, PhD and vice president of development at OpenEye Scientific Software, explains, “We have a huge C++ code base that is very template heavy. It had always taken a long time to compile, but Apple is the fastest compiler platform we have. Compiling something in 10 minutes versus 50 minutes makes a huge difference to us. We can use multiple cores to build pieces in parallel, and it is phenomenally faster than a high-end Windows PC. All of our developers want it because they can recompile and crank through—they don't sit and wait.”

The increased compilation speed, however, would mean little to OpenEye if Mac OS X wasn’t a robust multi-platform development environment. Their Tier 1 and Tier 2 platforms—which are defined by the availability of all current OpenEye software—include Microsoft Windows, Linux, and multiple flavors of UNIX, as well as Mac OS X.

To write clean code that can easily be compiled in each of these environments, OpenEye avoids using IDEs, sticking to traditional UNIX tools such as GCC and make. Their UNIX-brewed developers tend to prefer vi and Emacs for writing and editing code. Mac OS X, being based on UNIX, has each of these tools built in.

Tolbert says, “You can just drop into Mac OS X and you have Emacs and vi, you can build with make, you don't have to do anything special. You have very little chance of building code here that won't work somewhere else. If you sit in Windows for too long, you can write code that won't build anywhere else; on a Mac, you know that if you move it somewhere else, it is going to compile.”

Optimization Tools on a Stable UNIX Platform

If the availability of standard UNIX development tools and significant performance gains at compilation time hadn't already convinced OpenEye that Mac OS X was the premier cross-platform development environment, their experience with an Apple utility called Shark would certainly have pushed them over the edge. Shark is a profiling utility that is part of the CHUD (Computer Hardware Understanding Developer) tools, that help developers optimize code in Mac OS X. Many of the benefits of Shark will be seen regardless of what platform the code originated from or what platform the code is ported to.

Tolbert explains, “We've been really happy with the performance tuning we get with Shark, Apple's tool for optimization and profiling. When we get to the end and want to optimize an application, we run through Shark and every one of our command-line applications has gained a noticeable and measurable speed increase.”

Tolbert says his team currently works on multiple platforms, and uses a number of Macs along with Linux and other machines.

He adds, “You can do similar things with a command-line profiler in Linux, but it's much harder to dig through the output.”

Mac OS X 10.4 Tiger has ushered in some welcome changes for the developers at OpenEye as well. On Intel-based Macs, OpenEye is using GCC 4.0, which ships with Xcode 2.x. They have also found fixes and improvements in OpenGL—crucial for 3D modelling—that caused them problems on other platforms when users upgrade graphics drivers. OpenEye uses Qt in conjunction with OpenGL to further abstract the development environment from the runtime platform.

In fact, a packaged, supported 3D graphics solution has been difficult to find outside of SGI. Users who opt for hardware 3D stereo graphics in a Linux or Windows environment often find themselves dealing with driver issues that may or may not be solvable.

Tolbert explains, “You need a stable, set platform that you know works, as opposed to buying the machines from vendor A and special ordering the 3D card from vendor B and putting it together yourself to get it to work.”

An OpenEye View of Software Development

OpenEye doesn't currently cater to the novice developer or novice modeler. They target the key players at major pharmaceutical companies who both write and run the applications that make modern day drug discovery possible. In the pharmaceutical industry, successful drug discovery is highly correlated to the ability to logically pare down massive amounts of data into digestible subsets before real world experiments take place. This process is called virtual screening. To do this efficiently, you need the right software and the right hardware.

OpenEye's products break down into three broad categories: First, the computational chemistry software that is mostly command-line based and runs well across a cluster. Second, a large and growing set of toolkits written in C++ with Java and Python wrappers, making core components easily accessible to a wide variety of developers. Third, visualization tools and front ends, which scientists use to interact with their data.

Some of OpenEye's most popular products include OMEGA, ROCS, OEChem, and VIDA. OMEGA and ROCS, used in tandem, fit into the first category. OMEGA prepares a database to search and generate multiple 3D conformations for each set of molecules, while ROCS is a small molecule comparison tool that helps guide decisions.

OEChem is an extremely robust chemistry informatics toolkit written in C++, which allows pharmaceutical companies to leverage a solid base for writing their own proprietary applications.

VIDA is OpenEye's graphical interface that is used to “visualize, manage, and manipulate large sets of molecular information.” It is used by modelers with a set of results to share with a chemist, or teams of chemists. 3D visualization is core to the discovery process so hardware stereographic displays are the norm for the industry. Conference rooms with 3D goggles become virtual screening areas where scientists can almost literally climb around inside each molecule to get a better understanding of their results.

The Mac Pro is a premier workstation graphics platform with the optional inclusion of the NVIDIA Quadro FX 4500 graphics card. An integrated stereo 3D port makes it an ideal choice for the type of 3D visualization that is critical to running applications like VIDA.

Seeing Mac in their Future

Multi-platform development is important not just because companies use different platforms; mixed IT environments are becoming increasingly common, because of individual users shifting to different platforms. Employees often lead the shift to Mac OS X because they need the power of a UNIX-based operating system that provides an enterprise-class desktop, and includes the convenience and power of Microsoft Word, Excel and PowerPoint, all in one package. And you only get that with Mac OS X.

Mac OS X allows users to have the best of both worlds. As the pharmaceutical industry changes, the needs of developers and users are changing with it. As the big pharmaceutical companies are shifting away from older platforms, Apple continues to roll out increasingly powerful Intel-based systems running Mac OS X that fit their needs.