Speed up SOA/ESB software development with a "container-less" technique

Service-oriented architecture (SOA) and enterprise service bus (ESB) are two buzzwords that have gained significant momentum and traction in live architectures over the last few years. Speaking from firsthand experience, I would say SOA/ESB techniques do deliver increased modularity, flexibility and reusability of components, which has been the age-old promise of object-oriented programming (OOP) and even general software engineering. In particular, SOA/ESB appears to extend effectively on the hallowed OOP principle of encapsulation. But it seems all new paradigms come with a "dark side" that must be effectively recognized and managed to minimize undesirable consequences. What is the dark side of SOA/ESB? This article examines it and proposes a powerful technique and "quasi-architecture" as the solution.

The dark side of SOA/ESB is increased runtime and testing complexity. A complex SOA/ESB architecture can resemble a circuit wiring diagram, where service calls are the wires and operations are the components. However, circuits have an inherent one-way flow of electricity, and SOA can involve two-way communication (service A calls service B, service B calls service A). If you have n services, there are n-squared possible interactions between those services. Even limiting a system to some small subset of that total can lead to a tangle. Also, often later into development, one might realize that a particular function covered by service A is better handled by service B. Depending on the design, moving (refactoring) code between service boundaries can be difficult, time consuming and even fraught with peril. Is there some way to minimize all that?

The container

Outside this custom-built complexity is the problem of the container. For this article, "container" refers to a Java Platform Enterprise Edition (Java EE) Web server that supports Java Message Service (JMS) and, say, message-driven Enterprise JavaBeans (EJB), the basic and increasingly standardized enterprise approach to SOA/ESB. Two containers I have worked with are BEA WebLogic and JBoss. (They have properties similar to other containers, such as IBM WebSphere.) Despite all its power and justifiable fanfare, the container has a dark side too. I worked on a system involving just a half-dozen services, but noticed the following (note that many of these issues are also associated with simple client-server applications but are exacerbated with SOA/ESB):

• Long startup and shutdown time for the container. At several seconds per service, startup time for all services was nearing two minutes.
• Difficulty in debugging one service independently of another. If one service B is incorrectly compiling or behaving, it can be difficult for another service A to function, even if service A does not depend on B.
• Complex classpath setup requirements sometimes interfere with correct functioning.
• Complex container setup requirements. A distribution directory must be set up with the right jar files, Spring/Hibernate configuration files, etc. It is neither simple nor automatic to keep the distribution directory synched up with code changes.
• On Windows, (re)compiles/builds can fail because the container is "holding on" to files such as log files or jar files; i.e., a delete or overwrite of a file fails because the file is open by some process. Hence, the container often must be shut down and restarted just to do a rebuild/redeploy of the code.
• Using the debugger generally requires setting up a remote debugger. A developer must interrupt the process of running through code and frequently connect/disconnect. New deployment requires disconnecting/reconnecting the debugger. Many developers never set up remote debugging and resort to crude System.out.println and — not intentionally, but in practice — long edit-compile-test cycles.
• Since the container is inherently multithreaded, the debugging view is significantly complicated. WebLogic immediately spawns dozens, or more, helper threads on startup even if none are used immediately, or ever. This requires initialization and memory, and dormant threads seem to slow JVM performance on active threads—i.e., a significant waste of resources for "low load" situations.
• Container-based debugging typically involves scanning log files, which can be time consuming and problematic for many reasons:

oToo much output
oMust restart the container to make log4j logging-level changes
oLogs "scroll" to new files, and a simple scan does not keep up, thereby seeming to falsely indicate a code hang
oDifficult to understand what the system is doing if there is no log output
oMust recompile lots of code just to make minor new logging calls
oCannot access critical internal variables and stack state

All of the above interfere, sometimes in a time-consuming way, with the critical edit-compile-test loop, where developers spend most of their time. This overall cost is not obvious, difficult to quantify, and almost invisible. But at the end of the day, it can all add up substantially. Developers do not always realize how much time they spend setting up the system just to get the code to run, because it is sometimes a mechanical, almost-unconscious detour.

Can one mix SOA/ESB with RAD, or rapid application development? The above enumeration of the quite substantial "dark side" seems to preclude it. The container (and Java EE at times) could be said to suffer from "monolithicity."

Container vs. container-less

Contrast the massive overhead described above to the fast approach of running code in the Eclipse environment. The developer can just right-click on a Java file and run it as an application or as a JUnit test to instantly start executing the code — just point and click, or plug and play.

Eclipse always automatically compiles the code. And it supports complex inter-package dependencies. This mode of execution is sometimes called "container-less," to contrast it with its complementary mode. The big question is: "Why is executing code in the Eclipse environment so quick and easy, as compared to executing code within the container?"

The answer is clearly that the container provides many subsystems that "wrap" the code in various layers of insulation, so to speak. The major capabilities provided are:

• Concurrency: Any number of message-driven EJB components can run simultaneously on one machine.
• Transactions: Atomic commit/rollback of the whole service call or parts of it.
• Pooling: Such as with threads or database connections.
• Distributed computing: The same EJB can be run on multiple machines.

For this article, let's call the general union of the above properties "scalability."

What is remarkable is how little of this container-centric functionality developers care about in just implementing specific business logic, generally the main point of value for custom enterprise development. Most business logic is not really oriented around any of the above areas. Thus, programmers must deal with two major layers of complexity: application logic and container wrapping.

If all containers are somewhat interchangeable, then a programmer's coding effort that frequently focuses on a container's general minutiae (say, for example, JMS message-passing infrastructure, or mock/test/simulation systems) can be repetitive and unproductive, or in other words, not valuable. Yet, on the multi-developer SOA/ESB projects with which I've worked, the developers spend significant, even excessive time on all the container wrapping.

This is part of the core of what industry analysts such as Bruce Tate (in his book Beyond Java) complain about in their criticisms of the complexity of Java EE. But if this is the case, how can we develop code that minimizes time spent on container wrapping? Is there some kind of architecture approach that can minimize the amount of time the programmer deals with container wrapping?

Container-less testing

An answer, model, prototype, and solution can be found in extending the existing technique of container-less JUnit testing. Developers, while developing and testing SOA/ESB-type code, may have JUnit tests that exercise some large part of a service independent of the container, using the Eclipse "Run as JUnit" Java File menu option.

Container-less tests executed in Eclipse or from Ant do not remove all the above pains and disadvantages associated with the container but definitely diminish many of them significantly. Container-less tests are sometimes not as easy to set up because all project effort is typically driven toward getting the container-based code to function correctly. But often the initial up-front effort is well worth it, because of the possible immense savings in the edit-compile-test cycle.

The immediate problem with the container-less test for SOA/ESB code is at the service boundary. What happens when one service calls another, as is the case for any nontrivial SOA/ESB system? One typical approach is to use stub code for inter-service calls. However, stubs can be time consuming to write and maintain. A whole set of parallel code must be maintained relative to live code. Often, to effectively or nontrivially test a service, nontrivial (dynamic) behavior from the stub is required. Test systems that attempt to mimic live systems can evolve into halls of mirrors, with programmers wandering, meandering, or finding themselves lost in them. A developer can get sidetracked building and maintaining complex stub/mock/simulator systems.

Container-less tests for various aspects of the system, e.g., live database calls (or Hibernate), are well known. The approach I advocate is something similar that might be called "container-less inter-service calls," which may sound like a contradiction, but is possible; the sophisticated technique is not widely known but has many advantages.

A foremost killer-app advantage is that virtually no time is required for recompiling via Ant targets, and no time is required for shutting down WebLogic, or restarting and redeploying. Eclipse recompiles the changed Java files almost instantly.

Another paradoxical advantageous property is that unit tests become nearly indistinguishable from integration tests. It is possible to write a thorough inter-service unit test that is in many ways actually an integration test. This pushes integration testing closer to individual developers and away from the larger time-consuming process of inter-developer integration, and thereby tightens the loop. Each developer participates in more of the complete, overall functionality, and there is less working in isolated silos, or less of the dreaded, project-killing heavy-pain-on-integration effect. (These concepts are explored by Fowler in his continuous integration philosophy)

Note: Container-less tests require excellent understanding of the differences in the way the container and Eclipse manage the classpath and delicate care in setting up inter-project dependencies in Eclipse. The Eclipse classpath behavior can be extremely complex and can lead to subtle problems in running the container-less tests because of the multilayered way it builds up the classpath.

JUnit testing can be extended beyond mere tests to the overall functionality of the system. Consider some SOA/ESB processes that span multiple services. Imagine just having all the code in Eclipse and running it such that the code never makes any inter-service JMS calls — it just moves "quasi messages" around in memory while exercising all the code that crosses multiple services. Why not? Let's call this quasi-architecture "container-less SOA." It comes with the basic realization that services and JMS are not necessary for exercising all the business logic. JMS just boils down to a distributed parameter-passing system between Java methods.

Example: Container-less SOA

Enough of the pitch! Let's work through a tangible example and a live code fragment. Consider a basic emerging SOA/ESB (inter-service!) design pattern that could be called "the assembler." The assembler takes orders for a product and issues requests for parts to different services. This is a fundamentally cross-service operation, whose correctness is difficult to verify piecemeal. One standard approach might be to create mock part-services that reply with mock objects (mock parts).

Instead, consider the container-less SOA technique that exercises entirely live code. To be specific, the part-services are Customer, Order, AssetManagement, and AssetDetail. The Assembly service starts with a service call to Customer, then one to Order, then three in turn to AssetManagement, then AssetDetail, one more to AssetManagement, and finally one more to Order.

The container-less SOA code fragment to completely run all that business logic without mock objects looks as follows (the code reflects the unrequired condition that the messages are sent and delivered on different JMS queue names):

public void sequence() throws CoreApplicationException, JMSException {
Thread thread = new Thread(new Runnable() {
public void run() {
CustomerMessageBean customerbean = new CustomerMessageBean();
OrderMessageBean orderbean = new OrderMessageBean();
AssetManagementMessageBean assetbean = new AssetManagementMessageBean();
DetailMessageBean detailbean = new DetailMessageBean();

customerbean.ejbCreate();
orderbean.ejbCreate();
assetbean.ejbCreate();
detailbean.ejbCreate(); // normally the container calls this EJB method

MessageReceiver receiver = null;

try {
receiver = new MessageReceiver(jmsServerUrl);

Message msg = receiver.receiveQueue("customer_queue");

customerbean.onMessage(msg);

// assembly bean then calls Order bean to find
// out about the customer's order

msg = receiver.receiveQueue("order_queue");

orderbean.onMessage(msg);

// assembly then makes 3 sequential calls to assetbean
// for digital assets. like say maps or images

for (int i = 1; i <= 3; i++) {
msg = receiver.receiveQueue("asset_queue");

assetbean.onMessage(msg);
}

// get some addl details on the assets. say map location info
msg = receiver.receiveQueue("detail_queue");

detailbean.onMessage(msg);

// asset rendering files. eg fonts etc
msg = receiver.receiveQueue("asset_queue");

assetbean.onMessage(msg);

// assembly then wants more info on the order again
msg = receiver.receiveQueue("order_queue");

orderbean.onMessage(msg);

// assembly then gets a final asset type
msg = receiver.receiveQueue("asset_queue");

assetbean.onMessage(msg);


} catch (Exception e) {
throw new RuntimeException(e);
} finally {
done = true;
}
}
});

// start the above thread/mock-container/beans that will
// service the assembly bean in a concurrent "helper" or "supplier" thread
thread.start();

// receive the assembly message to "start assembly"
MessageReceiver receiver = new MessageReceiver(jmsServerUrl);
Message msg = receiver.receiveQueue("assembly_queue");

AssemblyMessageBean assemblybean = new AssemblyMessageBean();
assemblybean.ejbCreate();

// in the next call, the "supplier" thread runs concurrently with assembly.
// assembly makes the requests and the thread supplies them in the exact
// order. (if assembly makes a different sequence of calls than expected,
// then a more sophisticated supplier thread is required.)
assemblybean.onMessage(msg);

// at this point assembly has sent and received its last request
// and the thread terminated immediately after it sent the last reply
}
}

Note that all the beans actually extend the standard Java ME JMS message bean class, MessageDrivenBean. The supply-part service code runs in a thread to simulate the concurrency provided by the container. It receives the start-assembly message at the bottom, which is sent to the assembly bean. The assembly bean sends out its requests to the other services, which are then handled by the sequence of code running in the thread.

The MessageReceiver contains mainly JMS-oriented code. Here, I advocate that the only stub, or mock, code be JMS-simulating code that passes the JMS messages around in memory (implemented in code in MessageReceiver)—instead of through a live JMS implementation—thereby completely bypassing the need for the container, at least during development.

The other minor implementation detail is that in Eclipse, the "part" services must be dependent projects of Assembly. Previously, the services could be compiled independently, and that should be maintained, but the test code introduces the new dependency.

Using the above code, massive amounts of cross-service business logic and functionality can be exercised in a way that is indistinguishable from the live, production system. No stubs or their expensive associated maintenance for the services are required. A system that would probably be tested piecemeal or through time-consuming container startup/shutdown can be tested directly, immediately. Developer integration of the Assembly service goes from a complex undertaking to something almost automatic.

The above arrangement is not a "mock-service" system, it is actually a "mock-container" system! It simulates the core, basic operation of the container in instantiating message beans and passing them the message data.

Note the fine print: This container-less SOA pattern should not be taken to extremes. For example, if the individual part-services are extremely time consuming to run in their execution time, then mock services would be preferred. However, in general, the goal of minimizing the mock code/objects required is definitely preferred from the development point of view. For example, maybe only some subsystem of the part-service requires a mock implementation. Hence, my personal guideline: Mock systems should be used to replace time-consuming systems but not to avoid the difficulty of executing live code associated with a non-container-less SOA approach.

New perspective: Container scalability revisited

All this might lead to the question: "What exactly is the container providing if much of it sometimes gets in the way of development?"

One key answer is scalability. To summarize the previous section on scalability, it seems that scalability signifies that the code continues to run well for much larger datasets, hiding, as much as possible, the hardware from the software, so to speak. Java EE provides the infrastructure for scalability in the form of concurrency, transactions, pooling, and distribution. For example, any number of message-driven beans can run on any number of machines.

The container delivers on its key promise of scalability by, for example, providing seamless multithreading (or similarly, hardware clustering) of the message beans. But this multithreading capability is often of no direct need to the developer, who, during code development, is usually more focused on getting the sequence of business logic correct. Similarly for transactions — the developer should focus on writing code that works correctly and then just wrap it with the transaction logic.

Therefore, the developer should work with small datasets that minimize the edit-compile-test loop and best-approximate the larger datasets seen in production — or, more specifically, work with small datasets that best cover the same execution-code-flow of larger ones. The proposed container-less inter-service pattern is a powerful tool for this purpose.

This technique has an influence on integration testing. In this newer approach, the chief purpose of integration testing is to determine how the code functions under higher loads (i.e., scalability), not to find logical inconsistencies in the code (i.e., non-load-related bugs), which will have largely been ironed out in unit testing.

Effectively, integration testing is redefined. Individual developers have the full system at their fingertips. The focus of inter-developer code testing moves from standard integration testing (making sure pieces fit together) to scalability/load testing (pieces are already resolved to fit together; make sure pieces run fast enough).

Another way of looking at the technique is as a system for minimizing developer overhead. A system can be easy or hard to develop on. Minimizing developer overhead indirectly contributes to minimizing application overhead and ultimately decreasing overall project overhead.

High pliability via scalability plug-ins

I call this concept of container-less SOA a quasi-architecture because it can hide behind the real architecture and be visible mostly only to developers. Overall, it clearly leads to a new architecture concept. Instead of building a system in which all the full-blown scalability systems are entrenched in and tightly coupled with the code, the scalability systems can be utilized as plug-ins, and the developer can easily switch between the dual environments of low scalability and fast development to high scalability and heavy load testing. The simple philosophy expressed and manifested in the technique is that instead of the code being subservient to the container, the container(s) should be subservient to the code.

A related area is the use of the HSQLDB database. HSQLDB is an in-memory database that can handle most aspects of a production database like Oracle. The database can be seen as another scalability system. On one system, when converting almost 250 persistence/data-layer test methods to run in-memory on HSQLDB instead of Oracle, I saw a speed improvement of almost three times. There are even some gains to be had by using a low-overhead system like MYSQL instead of Oracle, if the persistence code does not use many database-implementation-specific features.

Developers can sometimes quickly iron out kinks in business logic by utilizing the less-monolithic, lower-overhead, faster, low-scalability systems. The standard concept of out-of-container testing is an acknowledgement of this reality. Container-less SOA and these related offshoots simply build on it in a logical progression.

Consider a system with a plug-in architecture for messaging, the container (bean management), and the database, including connection pooling. All of these could run in-memory in the low-scalability scenario, but be replaced with production-scalable systems with a quick flip of a switch (i.e., implemented as a Java property). We could call such a scenario "highly pliable." There is really no inherent technical restriction or constraint against this elegant architecture. Actually, maybe high pliability defined in this way is as valuable as other recognized critical features, such as high scalability. In actuality, they are tightly coupled with each other in successful systems.

Pliability of a system does seem to be orthogonal to the property of scalability in the following sense: Some systems are highly scalable in the sense of handling massive data loads, but have low-pliability in the sense of one developer not being able to either wrap his local effort around the overall system or exercise the overall flow of the system. Yet the latter is exactly what is required for maximum programmer productivity and effectiveness against that system. Maybe pliability is the opposite of monolithicity.

This view also meshes closely with the new widespread concept of dependency injection. The scalability systems can be dependency injected into the code. But that's another article!

The general direction of container-less SOA is a packaged architecture. The code could be developed such that it is one cohesive application that doesn't use any external systems, e.g., JMS/socket-based messaging, external database, connection pools, etc.; then a packaging step can substitute in the components that permit scalability.

With the container-less SOA approach, an added benefit is that programmer emphasis on developing container-common features, such as JMS libraries, is decreased. Also, because the message-passing architecture associated with container-less SOA tends to be better encapsulated, refactoring the code between services becomes more feasible. Once the code is refactored, it is easier to retest the full-flow functionality to ensure the refactoring did not change any logic, compared with a piecemeal testing system.

In fact, the whole concept of refactoring might be revisited under the container-less SOA architecture. The SOA architecture encourages building all the business logic to function correctly and then, at the very last moment, exploring, deriving, and delineating the service boundaries, which is quite the opposite of the current convention of attempting to dictate the service boundaries before any code is written. Putting business logic into services then becomes more of a final wrapping step rather than a cumbersome priority always to keep foremost in mind during development.

To summarize, a container-less SOA/ESB technique holds significant potential for speedier SOA/ESB software development. The SOA/ESB approach is a highly enterprise-level model right now and is somewhat far from utilizing a rapid application development scenario; however, the container-less approach brings these paradigms closer together. It uses the leverage of a self-contained system.

When AMD delivers a chip, its entire complex functionality is encased in a few-inch-square piece of packaged silicon, despite containing many sophisticated subsystems, such as a floating-point unit, a cache, or a logic unit. Similarly, with this approach, all the live inter-service code and functionality is almost fully circumscribed, available, and accessible in an individual developer's local environment, in addition to being spread out over the enterprise network. The developer can switch between the two with a flick of the switch for maximum coding and testing leverage. Container-less SOA contains a developer-centric philosophy at its heart, but not at the expense of the overall enterprise. Both benefit.