Special Reports

Integration at the Edge

We can learn lessons from trading hubs and apply the data to IT integration. See how the ESB removes the distinction among internal and external networks for supply-chain apps.

• By Dave Chappell
• 07/01/2004

Services-oriented architecture (SOA) has been held out for nearly two decades as a substantially more cost-effective and flexible strategy for constructing enterprise software systems than historical approaches including monolithic system design and tightly coupled client/server models. Many enterprise CIOs and industry observers believe Web services technology, and the unprecedented universal vendor support of the underlying standards, will finally make practical the widespread adoption of the SOA approach.

SOA Improves Business Responsiveness to Market Change – The services-oriented approach to enterprise software architecture replaces large, complex, monolithic applications with applications composed of loosely coupled collections of modular software components linked through well-defined Web service interfaces.

For the business, the result is movement toward the "real-time enterprise." Because applications have well-defined, service-based integration points, and because these services are built on standards embraced by every major enterprise software vendor, connecting software systems together will be orders of magnitude easier than in the past.

The ability to move the right information, to the right people and systems, at the right time, maximizes the ability of the enterprise to identify and interpret changes in its markets and to respond by adjusting its processes, operating models, or structure.

SOA Improves IT Responsiveness to Business Change – The business response to changing market conditions often calls for modifying, integrating, or creating new applications and systems to run the business. For the IT organization, the result of SOA adoption is a decrease in the cost and a dramatic increase in the velocity with which these changes can be made.

Cost is reduced in the application development cycle because existing application modules and services can be leveraged as building blocks, eliminating the costly and repetitive exertion of effort that often slows today's initiatives. Velocity improves because application development is more like "application assembly" when building on existing services, thereby slashing development time.

Enhancing the ability to change the enterprise application landscape cost effectively and rapidly, in support of changing business requirements, is precisely why organizations are adopting the SOA approach.

The Nature of an Enterprise Services Network
Enterprise services networks are, by nature, highly dynamic. Because change is made easier, new systems are completed more quickly. The applications of acquired companies are integrated more rapidly. Customers, suppliers, and partners are connected with greater speed.

Enterprise services networks get big rapidly. The number of application components in a services network is much larger than the number of applications in an enterprise where monolithic applications reign. The services-oriented approach breaks monolithic applications into smaller, more focused application modules that are easily understood, modified, and linked to form and reform applications as the needs of the business change.

Enterprise services networks are also characterized by complex interdependencies. In the past, monolithic applications had very clear boundaries. In a services network, "applications" have fluid boundaries; they overlap and they are often assembled by leveraging components and services provided by other organizations within the enterprise, or even across enterprise boundaries. Dependencies can be direct, indirect, and circular. Application modules often both consume and provide Web services. Applications are assembled by leveraging services provided by modules that themselves may rely on services provided by other modules, and so on.

The Impact of Change in an SOA
What do you get when you combine change, big networks of application components linked together in complex chains, and interdependencies that are often unclear (other than a migraine)? You get exploding costs associated with keeping the network running and meeting performance expectations—all while continuing to facilitate change (the reason for embarking on an SOA strategy in the first place).

As a services network grows, the marginal cost of change in the network grows at an accelerating pace. Three types of planned change predominate these rising costs: to a service in the network, to many services in the network simultaneously, to a service in the network unexpectedly.

Planned change to a service in the network – In many cases, intentional IT actions may cause unintentional impact to the services network. These intentional actions can include introducing new applications, replacing and enhancing existing applications and Web services, upgrading systems, bringing systems up and down for maintenance, relocating servers, and outsourcing applications. Here's an example of intended change with unintended consequences:

• Change: An aircraft manufacturer creates a new portal that will display real-time manufacturing process status. This portal is to leverage a Web service that returns the current status of a given aircraft assembly project. The Web service is changed to accept a more detailed query and to return a more detailed response.
• Result: Service contract changes.
• Impact: Systems bound to the old version of the service suddenly find their message formats invalid. Some systems that relied on the old version of the service were not even known to the Web service owner. A developer in another group had discovered the availability of the service over a water cooler conversation and had built a system that relied on the service, albeit infrequently. Discovery and quick utilization of a service that enables the quick creation of a business-enhancing application is a big benefit of the SOA approach. But that system is now a time bomb waiting to detonate the next time it tries to access the service through the old contract.

Planned simultaneous change to many services in the network – In some cases, change must be applied to the entire network of Web services (or some large subset thereof). Consider this example:

• Change: A commercial bank has added a new set of affiliates to its partner network. These affiliates will integrate some of their loan processing systems with those of the commercial bank. The bank makes this process possible through a collection of Web services that are provided by a number of different systems: some packaged applications, some custom systems, and some processes that are managed by an EAI platform. To secure these services, access control is enforced based on credentials passed in the Web service requests.
• Result: There are many systems that must be updated to ensure partner access is granted.
• Impact: The variety of systems providing Web services means multiple security frameworks must be understood. Adding partner access to a packaged application is different than adding access to a custom application, which is different than adding access to the EAI-hosted process triggers, which is a costly exercise both directly and in terms of the risk that is introduced as each system is touched and configured in its own unique way. While adding new applications to the network or modifying the behavior of individual application modules is made much easier through the SOA approach, it can be more difficult to make changes where global policy is involved.

Unexpected change to a service in the network – In a growing network of connected applications, service changes can occur suddenly and unexpectedly. Consider this example:

• Change: A manufacturer faces a sudden increase in market demand for a product. Traffic increases to an e-commerce site as customers place orders. More calls come into the call center where customer service representatives take product orders. Both of these systems link to a customer management Web service being provided by a CRM system.
• Result: An unexpected decrease in the response time of the customer management Web service.

Lets take a closer look at how unexpected change to the services network can cause a ripple effect of change throughout the services network, eventually impacting systems that are several times removed from the initial problematic service. These types of change include:

1. Market condition change: The problem starts with a change in a market condition—there is an increase in market demand for a company's product—which leads to an increased utilization of the e-commerce and call center order-entry applications. These applications leverage the customer management service provided by a packaged CRM system. As application utilization increases, so does the number of messages sent to the customer management service.
2. Impact to the services network: The increase in message volume to the customer service management services degrades the performance of the CRM system. The result is a decrease in response time and throughput of the customer management Web service.
3. Impact to inked applications: Any application that depends on the customer management Web service will now begin to experience performance degradation.
4. Ripple-effect services network impact: Any upstream service that depends on the (now performance-degraded) customer management service may now itself display performance degradation in response time. In this case, the EAI system providing the partner accepts the order process and the Web service trigger is impacted.
5. Impact to linked applications: As before, any applications that rely on the accept-order process to trigger the service (that relies on the customer management service) will now begin to experience degraded performance.
6. Continued ripple effects impact the services network and applications: Ripple effects continue until there are no more upstream systems acting as service providers. The result is an overall decrease in service levels for all dependant Web services within the network.

While this series of events paints an ugly picture, what IT experiences is even uglier. There are no lines painted on the floor of the data center highlighting the interdependencies among the loosely coupled systems in a services network. There are no performance gauges and no alarm bells that sound when service performance moves outside normal bounds. There is no means of tracing failures back to the root of a services network problem.

Instead, those responsible for keeping the services network up and running face a bewildering set of seemingly coincidental system failures. These failures are expensive as orders are lost, customers turn to competitors, and valuable IT personnel are troubleshooting problems frantically. As the services network grows, costs explode; the impact of a services network change ripples through a larger network of business applications.

Skyrocketing Costs: The Ultimate Impact of Change
While change is easy and affordable with an SOA, the impact of change (unexpected or planned) can be unmanageable and expensive. Rather than freeing IT to be more responsive to changing business needs, the SOA approach can quickly lead to a reduction in responsiveness and increased costs. To truly reap the benefits of the services-oriented approach to application architecture, organizations must have a way to manage the impact of change in their enterprise services network.

Web services management platform requirements – A Web services management platform is critical to the continued profitable growth of a services network. To do the job, Web services management products should address the three types of change in an enterprise services network that dominate rising costs, and provide a facility to:

• Know there is a services network problem – that the network of services is deviating from its normal operating range;
• Identify the root cause of the problem – if it is a services network problem, which application, service, and operation is at the head of the cascading chain of trouble;
• Insulate the downstream applications from the root failure – minimizing ripple-effect costs and providing breathing room to address the underlying problem;
• Understand the makeup of the services network – the true dependencies between systems, even those that leverage a service infrequently;
• Predict the impact of a change – will this new application overload the Web services to which it will connect? If I move an application providing services, which applications must be updated to interact with the new service?;
• Formulate the appropriate behavior – such as permitting or denying access to services based on the sender's identity, or sending billing information based on service utilization;
• Distribute policy, rules, and behaviors – into the network of Web services in a consistent and automated way.

In addition to offering these capabilities, a solution must scale and perform in a complex enterprise IT environment that requires these characteristics:

• Active – The solution must actively participate in shaping the behavior of the services network: redirecting traffic, changing in-flight documents, and enforcing policy.
• High performance – Solutions that can actively manage a services network must be able to manipulate in-flight XML documents efficiently. Efficiency results in low latency, high throughput, and fast service-response times. Horizontal and vertical solution scalability must be demonstrable.
• High availability – A solution must not introduce a single point of failure into the services network, and each individual component must be designed for maximum uptime and rapid failure recovery.
• Vendor neutral – Every enterprise-computing environment is heterogeneous. A solution cannot limit the types of solutions that can be actively managed in an enterprise-services network. A good solution will enhance an organization's flexibility in that regard.
• Nonintrusive – A solution must not require that systems participating in the services network be aware of its presence. Any solution that requires service providers or consumers to knowingly participate in the solution (by, for example, coding to a vendor-proprietary management framework) effectively creates a proprietary network of services.
• Deployment Flexibility – Solution components must be deployable on the widest variety of platforms and in the location most appropriate for the situation.

Architecting a Solution
Addressing all of the issues outlined here requires a powerful, flexible multitiered architecture like that provided by Actional and a variety of other Web services management vendors. This architecture must combine the power of centralized visibility and policy management with distributed monitoring and policy enforcement. In this approach, active components live in the enterprise services network, logically positioned between the providers and consumers of Web services, which Actional refers to as "in-network" components. A centralized management and policy server, working in concert with service registries and identity management solutions, communicates with these in-network components both receiving and sending information. Information received from in-network components includes service performance statistics, alerts, and service interdependency information. Policy, in the form of rule sets, is sent to the in-network components. These rules define in-network component behavior—when to raise alerts, how to process transiting message traffic, where to route messages, and how to enforce security policy.

Successive deployment of in-network capabilities for each new Web services project results in a fabric of control woven into the enterprise services network. The central policy and management server can then tap into this in-network fabric of control.

This solution provides a complete answer to the challenges faced in dealing with change in an enterprise services network, both unexpected and planned. It also addresses cost issues, as a managed enterprise services network automates the ability to respond to and embrace the constant change (planned or unexpected) that characterizes the enterprise services network—without the skyrocketing costs previously accrued.

Those enterprises that deploy a Web services management platform in support of their adoption of a services-oriented enterprise software architecture will get what they expect—movement toward operation as a real-time enterprise and an IT organization able to rapidly respond to and even enable changes in the operation of the business.