The decisions you make up front about hardware, site design, and operational policies can make the critical difference in your Exchange installation.

Best Practices of Exchange Planning

The decisions you make up front about hardware, site design, and operational policies can make the critical difference in your Exchange installation.

• By Kevin Kohut
• 02/01/1999

Ask any real estate agent what the three most critical factors are to consider in buying property, and the answer you’ll get is, “Location, location, location.” Likewise, the most critical factor of a successful Exchange deployment is planning, planning, planning.

Proper planning is important to any technology project; for a Microsoft Exchange Server deployment, it’s essential. Many aspects of Exchange Server are difficult to change after the fact (try renaming the organization after you’ve installed a few servers in the site). If things aren’t configured correctly, Exchange suffers performance degradations at the very least, and at worst, you can expect data loss accompanied by hours of rebuilding servers.

As I’ve worked on many Exchange projects, I’ve discovered several best practices—key items that are essential to successfully deploying and maintaining Exchange. By no means comprehensive, this collection of proposed hardware configurations, site design recommendations, and policy suggestions can save your headaches while ensuring a successful Exchange deployment.

Server Hardware

Under-powered or incorrectly configured servers account for most of the Exchange problems I’ve encountered. Exchange adores RAM, it thrashes hard drives, it pounds CPUs, and it loves stuffing packets through the network wire. If any of these areas are lacking, they can contribute to performance degradation. Upgrade one component, and Exchange will hit a bottleneck somewhere else. A holistic approach is definitely in order here.

Double Up Those Processors!

Let’s start with the processors. Obviously, the faster the CPU, the better the performance. But since Exchange is constantly processing short, I/O-intensive tasks, like writing transaction logs to disk, you’ll see even better performance with multiple processors. You’re better off with a system running two CPUs at 200MHz than a system running just one processor at 450 MHz. There’s a point of diminishing returns, however. Even though Windows NT can support up to eight CPUs, Exchange can’t do much with more than four. I typically recommend a dual-processor Pentium Pro 200MHz system.

Get Greedy with RAM

As for RAM, don’t be stingy. Use 128M as an absolute minimum. I’ve recommended no less than 256M to my clients, and because RAM is so inexpensive these days, I’m starting to push for 512M, or even a gigabyte of RAM! The more physical memory that Exchange can use for data, the less it has to write to disk. You can’t put too much RAM in an Exchange server.

The Right RAID

To fully appreciate how Exchange uses disk drives, it helps to know a little about how data is processed. When a user sends an e-mail message through Exchange, the system first writes a transaction log to disk. Note that the actual information store database isn’t accessed at this point. When the system has some idle CPU time, the transaction logs that have accumulated are written to the actual database. This method of saving data provides both fault tolerance and performance, but only if the disk drives are configured correctly.

Because transaction logs are written sequentially to disk, they benefit most from a single spindle drive configuration, dedicated for the log files.

Log files are also crucial for recovering from a system failure, so they need a high level of fault tolerance. A hardware RAID 1 set-up fits the bill nicely. It mirrors the data, providing excellent fault tolerance, and each disk in the mirror set can write the data fed to it using just one head.

The information store databases, on the other hand, are written randomly. They also can get quite large, typically 15G or greater. A RAID 5 configuration, with as many physical drives as possible, provides the best performance for the database files. For even faster performance you can configure write-back cache on the RAID 5 system to 100 percent (in other words, all write and no read caching). Do this only if your array has reliable battery backup for the cache—if you lose the data in the cache, the transaction logs are impossible to replay after a crash!

Finally, the NT operating system should be on its own physical disk, preferably mirrored for fault tolerance. This disk will also store the paging file, so make sure it has lots of space.

Network Needs

An often-overlooked component of an Exchange server is the network interface card (NIC). All communication between Exchange servers within a site is handled through Remote Procedure Calls (RPCs). These RPCs have a voracious appetite for network bandwidth. To avoid bottlenecks at the NIC, make sure your servers have NICs that are at least as fast as the network backbone. I like to put in nothing slower than 100 Megabit Ethernet.

Table 1 shows an Exchange server configuration that I can feel good about recommending.

Site Design

Ask three Exchange gurus how to design a site optimally and you’ll get three answers. It’s true that creating an architecture for an Exchange environment is more art than science; however, some important issues—dealing with server roles, replication, and site connectors—should always be considered.

Exchange does more than just deliver mail. It manages communications between sites; it handles Internet mail; it keeps public folders in sync among all the servers in the site; it expands global distribution lists into individual mailbox addresses; and each Exchange server has to keep tabs on all the other servers in the organization. Many of these functions can be relegated to a specific server, reducing the load on other servers. The idea is to have two types of Exchange servers: those that handle user mailboxes and those dedicated to certain other tasks.

The Job of the Mailbox Server

Mailbox servers should be just that—they should run only those services necessary for handling Exchange mail: Directory Service, Information Store, Message Transfer Agent, and System Attendant. Mailbox servers shouldn’t be configured with site connectors or as Internet mail gateways. I also like to keep public folders off servers that have user mailboxes. This isn’t always practical, as it requires additional servers. (More on this in a moment.)

If your Exchange infrastructure includes Internet mail connectivity, you should set up at least one Internet mail gateway. This machine will run the Internet Mail Service (IMS), and be dedicated to processing SMTP mail. IMS machines can be configured to only handle incoming or outgoing mail, but they don’t support true load balancing. In most cases, one machine is enough.

In a multi-site environment you’ll need to set up at least one connector between each site. You should do this using bridgehead servers, which are machines dedicated to running site connectors. You can run more than one connector on a bridgehead server but, depending on the type of connector, you might have to modify some registry keys (this applies to systems running more than two X.400 connectors).

Another function you should offload from mailbox servers is distribution list expansion. When mail is addressed to a global distribution list, Exchange must first expand that list into individual mailbox addresses. A single e-mail message, sent to a list with 200 addresses, can bog down several servers as Exchange tries to figure out which server each address belongs to. Configuring large distribution lists to expand to a particular server helps ease congestion and allows quicker delivery of the other mail queued in the MTA.

Public Folder Particulars

Let’s get back to public folders. Every server within an organization that has a public information store automatically sends one or more status messages to other public folder servers every 12 hours. These messages update other servers on changes that have occurred within the public folder structure since the last status message. Even if a server’s public folders are empty, it still sends the status message and receives messages from other servers. To keep this status message traffic at a minimum, remove the public information store from every server that isn’t actually using it. You can do this through the Exchange Administrator program, by simply highlighting the Public Folder object of the appropriate server and then pressing the Delete key. You’ll get a warning message, but once you delete it, all the data is gone for good (unless you restore from a backup tape).

Public folders send more than just status messages to each other; they also send replication messages. By default, these messages are sent every 15 minutes. You can cut replication traffic by increasing the default interval. You do this through the Exchange Administrator program. First, select the server you want to configure, then double-click the Public Folder object to bring up the Properties window. Next, click the Advanced tab, and change the replicate always interval to a higher value. The time interval you choose will depend on how often public folders are changed; but make it at least 30 minutes. Unfortunately, you’ll have to configure servers one at a time.

Operational Policies

Exchange affords many ways of controlling its operation, from setting limits to backing up data to working with other mail systems. In many cases these options are available at various levels along the organizational hierarchy. Mailbox limits, for example, can be set at the individual mailbox level all the way up to the entire organization.

The two most important limits to establish are mailbox size and message size. Setting proper mailbox limits ensures that you won’t run out of disk space unexpectedly. This is critical, because Exchange will freeze a system if it can’t write to the information stores. Exchange can also freeze when the MTA attempts to process too large a message. That’s why limiting large attachments is also critical.

Message Limits

The best way to keep users from sending huge attachments that can choke the server is to enforce message size limits. Based on real-world observations, I’ve discovered that as messages (which include attachments) approach 15M, the MTA starts to have troubles. I recommend that 10M be the absolute maximum. For Internet mail—that is, mail being handled by the IMS—it’s good to go even lower. I usually suggest 2M. Users may not like this restriction; their friends won’t be able to send them the latest video clip. I counter this with a short lesson on how to use FTP to transfer files.

Figure 1. Limiting message size is crucial to keep Exchange from freezing.

Mailbox limits

Once you’ve figured out message size limits you can then determine the mailbox limits. There are three types of mailbox limits that can be set: a “soft” limit that simply generates an e-mail to the offending user, and two levels of “hard” limits. The first hard limit prevents the mailbox from sending mail, the most restrictive hard limit prevents the mailbox from sending or receiving mail.

Figure 2. Mailbox limits help prevent running out of disk space.

Ideally, you’ll determine these limits before you spec out the server hardware. Then all you have to do is multiply the most restrictive limit by the number of mailboxes you plan to put on that server. Now use that figure to determine the total disk space you’ll need for the private information store. If this server is going to contain public folders, make sure you allot some space for the public information store. For example, if you want to put 600 users on a server and you’ve decided on a 40M mailbox limit, you’ll need 24G for the private information store. If you anticipate the public store to store around 2G of data, you should start with a 27G stripe set for the databases; I recommend adding at least another 4G to accommodate growth.

Unfortunately, server disk space is usually determined and installed long before mailbox limits are considered. In this situation, it’s not that complicated to calculate limits working backward from total disk space.

First, figure out how much space is available to the private information store. Start with the total stripe set, and subtract what the public store is using. From that figure, take off a few more gigabytes to leave some breathing room. Next, divide the available space by the number of mailboxes you plan to put on the server. Let’s say you have an Exchange server with 700 mailboxes on it. This server also has public folders that contain about 12G of data. The RAID 5 stripe set has 54G of total disk space. Subtract the 12G that the public store is using from the 54G total, which will give you 42G. Take off another 7G to allow for growth, and that leaves 35G available for the private information store. Divide that by the 700 mailboxes, and you end up with 50M per user.

Now that you’ve determined the send and receive limit, you can figure out the other mailbox limits. A good rule of thumb is to double your message size limit, and use that figure to stagger your mailbox limits. For example, if your message size limit is 5M, double that to get your stagger amount of 10M. So, if your mailbox send and receive limit is 50M, you’d set your send limit to 40M and the soft limit to 30M.

Backing Up

Mailbox and message limits help prevent problems with Exchange. A proper backup strategy ensures that you can recover data after a problem occurs. Everyone knows the need for routine backups, and most Exchange environments I’ve seen have a backup process in place. But just dumping files to a tape drive won’t guarantee that Exchange data can be recovered.

You can back up Exchange data either through an on- or off-line process. An off-line backup is simply a file-level copy to tape. Exchange services must be shut down for an off-line backup. And if you’ve got a large information store, it can take a couple hours to complete. The biggest problem with an off-line backup is that the Exchange databases may not be in a consistent state. Restoring data from an off-line backup of an inconsistent database requires that all the transaction logs and check files are restored. If any of these files is out of sync or otherwise corrupted, you can’t be sure of accurate data.

An on-line backup uses Exchange services to get to the data. Exchange isn’t shut down, and users can access their mailboxes without interruption. But the best advantage of an on-line backup is that it ensures the databases are consistent. You don’t have to worry about transaction logs or check files. The backup software you use must support Exchange to do on-line backups. When Exchange is installed, it automatically patches NT Server’s built-in backup program, NTBACKUP, so that it supports on-line backups. ArcServe, Backup Exec, and other popular NT backup software also include agents for Exchange on-line backups.

Even if you’re using on-line backups, an unexpected system crash in the middle of the day may not be recoverable. To survive such a catastrophe, you must disable circular logging on Exchange. To understand how circular logging works, let’s revisit how data is processed by Exchange. First, Exchange writes data to transaction log files. These files are always the same size—Exchange creates new log files as needed. Exchange also maintains a checkpoint file, which contains information on which transactions have been committed to the information store databases.

With circular logging enabled (which is the Exchange default), Exchange overwrites existing log files after the transactions they contain are committed to the store databases. Disabling circular logging forces Exchange to always create new log files, rather than overwriting existing ones. This ensures that you can play back all the transactions since the last backup.

Figure 3. Circular logging is enabled by default. Make sure you turn it off to ensure data recoverability after a system crash.

Closing Thoughts

Exchange is a complex animal. To properly deploy it, not only do you need a solid understanding of Exchange itself, but you’ll also need expertise in Windows NT, networking protocols, and hardware design, as well as an understanding of the non-technical political issues. Remember: The three most important things to do in deploying Exchange are to plan, plan, plan!

• Plan your server hardware.
• Plan your site design.
• Plan your operational policies.

The best practices I’ve presented in this article just scratch the surface. But they’ll put you on the right course to build a solid framework for your Exchange deployment.