In-Depth

Orchestrate Complex Data Collaboration

Use BizTalk Server 2004 to orchestrate complex data collaboration and exchange scenarios.

• By Danielle Ruest and Nelson Ruest
• 08/16/2005

The Merriam Webster online dictionary gives two definitions for the word "orchestration." The first states that it is "the arrangement of a musical composition for performance by an orchestra"; the second, that it is simply "harmonious organization." Both apply well to Microsoft's BizTalk Server 2004, although the second might be more on the mark.

BizTalk is a product that is designed to orchestrate information exchanges between either homogeneous or heterogeneous systems. It's meant to support multiple orchestration scenarios through business process management (BPM), enterprise application integration (EAI), or business-to-business exchanges (B2B). These scenarios work in multiple environments because of the way BizTalk Server itself works.

The basic concept of BizTalk is that it exchanges messages from one system to another. If you have a system that stores your inventory and you have another system that manages client information, you can use BizTalk to link the two systems together. For example, when you're working with a client that wants to order a certain amount of product, you can have BizTalk send a message to the inventory system to find out the stock level of an item. If you place the order, BizTalk can automatically go into the other system and deduct the amount you sold from current stock levels. This is because one of the core concepts of BizTalk is the adapter.

By default, BizTalk includes quite a series of adapters—adapters for the hypertext transfer protocol (HTTP), the file transfer protocol (FTP), the simple object access protocol (SOAP), Microsoft message queuing, file transfers, the simple mail transfer protocol (SMTP), Microsoft SQL Server, electronic data interchange (EDI), and even custom adapters for IBM MQSeries as well as DB2 and the SAP business suite. Most of these adapters are free with the purchase of BizTalk Server, but some, such as those developed by Microsoft partners, require additional fees (see Resources).

This means that BizTalk is designed to integrate with almost any system in existence. The integration process is fairly simple and can go from a basic orchestration to a complex design that can involve a multitude of partners (see Figure 1). Everything relies on the core BizTalk engine. Like the conductor in an orchestra, the core BizTalk engine manages all the orchestrations among the members of the system. Orchestrations, based on business rules, are sent as extended markup language (XML) messages to the BizTalk message box. This lets the BizTalk server connect to orchestration members through subscriptions, which know which adapter to use for both incoming and outgoing messages. If further integration is required—for example, if you need to connect to mainframe systems to retrieve data—you can add Microsoft Host Integration Server 2004.

In the end, the process is relatively simple. If you need to have two systems or more talk to each other, then you use BizTalk to do the conversions for you. It can take a basic message, transform it from its original format into XML, process it, and then send the result to another system, once again transforming it in the proper format. This is business process management, in which BizTalk harmoniously orchestrates exchanges from disparate systems and organizes them into a single, integrated workflow.

One other enticing aspect of BizTalk is the business rules engine. According to Scott Woodgate, lead product planner in the Microsoft Business Process and Integration Group, "this engine takes the pain out of business process changes... That's because when a business process changes, the orchestration itself doesn't necessarily change. The partners may be the same and the target systems may stay as well, but the way the process is consumed may have changed. With the business rules engine, all you need to modify is the business rule itself, not the entire orchestration. Once the rule is changed, the effect is immediate. This is a lot simpler than modifying the entire orchestration," he continues.

An Orchestration Example
The best way to see how BizTalk can help in orchestration scenarios is to look at an example. Let's use an example that relies on a criminal justice business process. Start by looking at the players (see Figure 2). Our players include all levels of the justice system: federal, state, and municipal, as well as other participants such as the police corps, correctional services, attorneys, prosecutors, and so on. This example is a good representation of the real world because each of the organizations involved has developed its own in-house system independently of the others. This means you'll find all sorts of systems in this group.

Larger groups such as the police corps, correctional services, and the justice department might have implemented proprietary database systems based on a variety of database engines such as Oracle or SQL Server. And perhaps some of the systems run on other, less well-known products. Smaller organizations such as the attorneys, notary publics, and so on might have implemented small business systems based on simpler technologies such as Microsoft Access.

Some organizations might not even be automated, which is an advantage in this case because you can dictate what they should get and make sure it is a standard implementation—in this case, an implementation based on a service-oriented architecture (SOA) would be best. But for the purpose of this example, every organization will already have an automated system in support of its mission.

A common scenario in this environment occurs every time a criminal infraction is committed. The police are called through the 911 service. They appear at the scene of the crime and start their investigation. This investigation eventually leads to an arrest, but they require a search warrant first. For this, they need to communicate with a justice of the peace. This communication is electronic and must be digitally signed. Once the process moves further, a prosecutor must be assigned to the case, a court date must be fixed, a judge must be assigned, and a public defender might even be required. If the defendant is underage, the youth authority might need to be involved. Because the incident occurred within a given municipality, municipal authorities need to be contacted. The same might apply for federal authorities and correctional services once or if the suspect is convicted. All these processes involve the exchange of information, ideally all in an electronic format (see Figure 3).

All the systems might run on different platforms and require different information formats. To create an integrated justice system, you might want to start development from scratch to produce a monolithic system where all information is in the same format. This would require the introduction of a new data center that will house this new system. Imagine the difficulties inherent in such a system creation. For one thing, the police corps will be most unlikely to trust anyone else with the management of their data. For another, the justice department does not control all the players in this system, so it is difficult to enforce standards. In addition, if you need to develop something from scratch, you probably won't see it in your lifetime.

If, on the other hand, you decide that what you need is a justice orchestration system, then you can turn to BizTalk to build a simple solution based on existing systems. First, identify which systems are in place and which ones need to exchange information. Several key players will have their own systems in place. This should include the police corps, correctional services, the justice department, and the court system—they might even have a digital courtroom recording system in place. Other players in this scenario will also already have systems in place, but let's begin with these first four because they have more regular dealings together. One of the partners, such as the justice department, could host a BizTalk infrastructure within its current IT environment, simply adding it as an additional service. Then, through tools such as the Single Sign On service available in BizTalk, you could ensure that the BizTalk adapters have the right to gather and exchange information between these core systems.

Microsoft rebuilt BizTalk from the ground up in this version to make sure it would be based on the .NET Framework, and added significant functionality to the product. For example, in large implementations BizTalk is now divided into a front-end and a back-end set of services. At the back end, you'll find SQL Server 2000, which can be hosted on a Windows Server 2003 server cluster of up to eight nodes. This cluster will store the state of your system—rules, subscriptions, configurations, and so on.

And at the front end, you'll have the BizTalk machines themselves running the various BizTalk services. These include only stateless information and deal only with the processing of exchanges. Normally in such a configuration, you'd rely on Windows Server's Network Load Balancing services to provide both load balancing and increased performance, but not with BizTalk. That's because BizTalk includes its own load balancing capabilities. Just add machines to the BizTalk system and they will automatically share the load among them. You can, of course, assign specific BizTalk roles to each server in the farm, but it isn't necessary because the service itself will be able to manage and control the load balancing of all services. You can find out more by viewing the High Availability Guide for BizTalk Server, available for download at Scott Woodgate's blog (see Resources).

All this means is that to implement an integrated justice workflow, you can start with the simple integration of a couple systems, improving the way the justice process works at the same time. Then, as time goes by and you get to be more familiar with the overall BizTalk capabilities, you can add more machines and more processes to the integration platform. Eventually, you'll have a system that automates and transforms data exchanges from all partners. The system is secure because it doesn't retain any of the information it processes; it only transforms it and sends it off to the appropriate target. Data is stored in the proper and appropriate databases and is protected by existing systems. There is no danger of data loss because the exchanges themselves can be secured through IPSec tunnels and other encryption mechanisms.

When you compare this to the traditional monolithic approach, you'll soon discover that the best thing about this is that you can start this system today, not a few years from now because you don't have to rethink the whole system. All you need to do is focus on some key processes and automate them.

This fits in with Woodgate's advice. According to him, any move to BizTalk server should use the "crawl, walk, run" approach. Begin by crawling, or connecting two disparate systems together. Use BizTalk for this instead of coding everything from scratch. Once you see the result, you learn to refine the business process rules you use for it and perhaps identify ways to improve the overall process. This is walking. Finally, as you get more familiar with the tool, you'll start working with BizTalk's Business Activity Monitoring tool and learn to refine your business process rules even further. This is running.

This is sound advice. You can't do it all at once, but one thing is certain: Whether you have an orchestration scenario that is as complex as the one described here or not, you'll soon discover that integrating systems through BizTalk is a lot easier and much more cost effective than doing it from scratch.