In-Depth

Exchanging Pain for Pleasure

Moving from Exchange 5.5 to Exchange 2000 is a complex task. This migration took many unexpected twists, turns and hours.

Exchange Server upgrades are an accepted aspect of Microsoft life. As Active Directory spreads its tentacles further across the network landscape, one of the natural steps is to move Exchange 5.5 servers to the latest AD-integrated versions.

It can be a difficult and unpredictable maneuver. The pressure was even greater in this case, as it was my company’s first non-test, real-life customer migration.

The Background
The customer’s network: Windows 2000 mixed mode, 26 workstations, four servers, one Exchange 5.5 server running on Win2K Server, 10/100 switched Ethernet, several network printers and various other devices spread around the office.

The client who wanted the upgrade has been a customer for several years. It has set office hours and usually only a few employees who work on the weekend; this makes for easy scheduled network downtime when needed.

The plan was to do an in-place upgrade of the Exchange server during a weekend. We conducted test runs outside the live environment and gathered documentation and caveats from the Exchange documentation and the Knowledge Base. We created backups and tested Thursday night, as well as immediately before we started the upgrade process Friday night.

We were ready. We were also planning on going home before midnight Friday. That isn’t what the Exchange server had in mind.

The Upgrade Begins
We met to start the process at 5:30 p.m. Friday. With the Exchange server offline, we performed the necessary tape backups. Verification showed the backup was good. As an extra precautionary step, we made backups of the Exchange database by stopping the services and performing a manual copy of all relevant files. This can save some time if you have to do a restore, as there’s no need to wait for tape.

As a first step, we disabled all incoming mail to keep from accidentally black-holing any e-mail. A quick firewall rule-set change stopped the incoming flow of e-mail.

Following the Microsoft upgrade procedures, we prepped AD for the Exchange upgrade. Modification of the schema is always a scary thing; but, in this case, we were extra paranoid since this was a production network. The prep went well, though. AD appeared to be healthy and we received no errors. As of 7:30 p.m., we were still on track. The upgrade on the Exchange 2000 Server then began.

The No-Fix Fix
After starting the upgrade process, we figured we had 90 minutes of sitting around watching progress bars slowly creep across the monitor. About midway through the upgrade process, an error requestor appeared on the screen. It told us there was an error updating the Internet Mail Service; we could retry or cancel. A Knowledge Base search revealed that Microsoft knew about this error. We thought we’d been saved.

But no—the fix we learned is that there is no fix. You must manually uninstall Exchange 2000 and the AD integration. The fix referenced by the Microsoft article involved removing registry entries, deleting files manually, restoring Exchange 5.5 from backup and attempting the upgrade again. So we started manually removing Exchange 2000 and restoring Exchange 5.5. Approaching 10 p.m., we were becoming thankful the customer had free soda for vendors.

After the removal process was complete, we reinstalled Exchange 5.5 and the service pack and restored the database to bring up the server. Following that, the store service wouldn’t start. We checked the Knowledge Base, copied files to and fro, adjusted settings, checked registry entries, copied files back and forth (again), reinstalled a couple of other times for good measure and restored files from backup—but the store was still dead. At 2 a.m. Saturday, after working the problem for a couple of hours, we made a decision not to troubleshoot the server anymore; but we didn’t want to destroy the current server before Exchange was fully functional. We wanted to get another Exchange Server online before formatting the old one for a clean install.

Moving to Plan C
In the corner of the room was a dual Pentium 120 Compaq Proliant Server with a RAID 5 array of four 2.5GB hard drives. It was this customer’s primary server for years but had been moved to a secondary test-server role. The ProLiant was running NT 4.0, the new server Win2K; we decided to try restoring the backup to the old server under NT 4.0/Exchange 5.5 in order to save time and not destroy the poor machine’s performance. The risk here was that Win2K would treat Exchange 5.5 differently enough that we would just end up wasting hours trying to restore the data to a server with NT 4.0 installed.

After installing NT 4.0, we installed Exchange and SPed it and restored the database. Deciding the backup tapes were probably the best option, we pulled them out and started the restoration. What we didn’t know was that the online backup from the previous evening went fine, but the backup performed immediately before the upgrade didn’t successfully pull the database. Hidden in the log file was a message that the Exchange offline public and private store backup failed, even though the backup reported it a success. The file-copy backup we did wasn’t as clean as an online restore, but it allowed us to regain the most current data. After starting the file-copy, we watched the status bar creep across the screen for a half-hour, holding our breath.

After the restoration was complete we did some minor testing. There were some oddities in the way Exchange acted, which was due to an Service Pack issue. We looked over our notes and discovered we were running the wrong SP for Exchange 5.5. So, we installed the proper SP, restarted the services and verified server operation. The Exchange database was resurrected from the dead and, like us, was breathing again. It was 4 a.m. when we went home.

Six hours later, we started planning the next action. Now that the old server was up and running, and Exchange and all had tested well, we could focus on getting the Win2K box operational.

Migration Good; Upgrade Bad
Reformatting the primary Exchange server, we installed Win2K Server and Exchange 2000. Once complete, we made a decision to migrate the data rather than attempt another upgrade. Now that a second server was operational and fully functional, migration was an easy solution.

Exchange 2000 came up cleanly on the server and appeared to be functional. Testing verified that it was running well, so we started the user migration. The migration went perfectly. The second step was to move the public folders. That went less perfectly. The machines reported that the migration was successful, but the folders never appeared on the destination machine. A run through the Knowledge Base and elsewhere on the Web revealed nothing, leading to more troubleshooting. Eventually, we used an older Exchange method of migrating data when all else fails: We exported all the public folders to a .pst file and imported them from a client machine. The permissions were lost, but the data was saved. That necessitated creating a list of necessary permissions and rebuilding them on the imported data.

We finished the configuration by configuring SMTP, modifying the firewall rules to allow mail to flow again and restarting the backup system. We were feeling good. After a bit of testing, we were ready to go home for the rest of the weekend.

The Aftermath
Over the next couple of weeks, some small issues came up:

  • Clients were receiving free/busy data update errors on their workstations. This turned out to be an Exchange permissions issue.
  • We noticed problems with Exchange addressing e-mail and relaying mail properly when used with our relay/smart host server. This was easily resolved through small configuration changes on the Exchange server.
  • We had problems with adding new users and a problem adding and modifying distribution lists. This was documented and repaired when we made the switch to Native mode.
  • Some of our Distribution Lists had newly illegal “*” characters in the names. After the conversion to Win2K, we had to adjust the naming convention on the distribution lists so we could keep a standard with the new limitations in addressing.
  • Personal Calendar permissions were broken because of some of our lists having the character “*” in the name. For a temporary fix, we added the users where needed, but after the switch to Native mode, we properly fixed the problem by creating all new distribution lists.
  • The transaction logging function of the database started filling the hard drives quickly. We developed a plan to create proper backups of log files and maintain the size of the directory over a longer term. If you aren’t familiar with the logging methods in Exchange 2000, I’d suggest you do some reading on it before you upgrade.

Most of our minor complaints and issues were resolved with the switch to Native mode. The other problems we had were repaired by the minor adjustments above.

Additional Information

We turned to the following Microsoft Knowledge Base articles for our project:

  • Q316886: How to Migrate a Exchange 5.5 server to Exchange 2000.
  • Q281456: The error that marked the end of a smooth upgrade (this was the first error that showed that the upgrade path was getting bumpy)
  • Q260378: Instructions on How to Manually Remove a Failed Exchange 2000 installation (It didn’t work for us; we still had to rebuild the server when all was said and done).
  • Q273478: How to Remove Exchange 2000 From Active Directory.
  • Q259712: Using the Move Mailbox method to move users to an Exchange 2000 Server.

Plus, you should also check out these resources:

—Alan Caruth

Learning Curve
Some of the lessons I learned from my experience:

  • When planning to install Exchange 2000, migrate, don’t upgrade. This may not apply in all circumstances; but if you have the opportunity and hardware to migrate, use that method. It will help you avoid most of the adventures we experienced in our first install.
  • Always verify that you have at least two good backups and can restore the data successfully. It’s much better to spend an extra hour making spare backups than spend dozens of hours explaining the lost data to the boss or client.
  • Create a backout plan. Occasionally bad things happen. When they do, be prepared. Test your backout method to verify it’s fully functional and will recover all data.
  • Check your permissions and do full end-user testing. We only discovered some errors after a couple days of production on the system.
  • Look in the Event Logs to find out, before the upgrade or migration, if you’re receiving any errors. A failed or misconfigured installation can create errors, which might not appear at first to be related to Exchange.
  • Do offline maintenance on your Exchange database before making the switch to Win2K. This can help compact the database and clean it up before you make the move, depending on how you do it. This also saves tape restoration time in case you end up having to back out of your installation.
  • Read the directions for the upgrade or migration. Read every ounce of them available from Microsoft and other resources. Follow the procedures exactly as stated. Look through the Knowledge Base for articles so you have an idea of what types of installation caveats and failures might occur. Familiarize yourself with the install process inside and out. Have the backout and uninstall procedures handy.
  • Plan the first migration or upgrade with a long maintenance window if available. If not, have a machine immediately ready to take the place of the machine you’re upgrading or migrating. If we’d scheduled our first upgrade for a weeknight, we would have spilled into regular business hours and cost the company money.

Remember Where the Exits Are
This was our first production Exchange 2000 migration. We’d prepared to a great extent before this upgrade and tried to take into account every possibility. Even then, an amazing number of unpredictable issues came up. The Knowledge Base, Exchange FAQs and discussion boards are great resources, but they can’t help in every situation. Be prepared to back out and do the upgrade another day, if needed. Migrate instead of upgrade and you will save yourself many headaches—and may even free up a weekend.

comments powered by Disqus

Reader Comments:

Fri, Jul 11, 2003 Network Creature Dominican Republic

Really great article!!! I know what he was talking about, I suffered a lot trying to migrate, so yes, let's go for the migration instead!!

Thu, Jul 3, 2003 Andy (been there) london

Well written, very real.

Thu, May 29, 2003 John MA

Been there, done that, ...don't feel so silly now!! ;-)

Wed, May 28, 2003 Anonymous Anonymous

Thank you, as we are looking into both methods. This has changed our mind!

Wed, Apr 23, 2003 Mike Qualls Seattle

Great story!! I wish I would have read it two weeks ago before the disaster of an upgrade I had. Same issues, same solution. Its amazing how we all tend to think alike when in disaster mode.

Sat, Apr 19, 2003 Anonymous Anonymous

I laughed for over an hour reading this. This is probably due to fear as my first migration looms

Thu, Nov 21, 2002 Gene Dallas

Really appreciate this story. Helped me change from upgrade to migrate.

Sun, Nov 10, 2002 Sergio ribeiro brasil

Very good

Mon, Sep 23, 2002 Joey India

Yea, we also faced a bit of problems during upgrade. But migrate helped to move to Exchange 2000. Here we have used bv-Admin for Exchange Migration utility to migrate all our Exchange resources to Exchange 2000.

Mon, Aug 26, 2002 Anonymous Anonymous

Thanks for the heads up! Soon very soon.

Sat, Aug 24, 2002 Jeff Brook Anonymous

Great Case Study!

Wed, Aug 21, 2002 Rich Connecticut

Boy, glad our first attempt last month went smoothly in only 6 or so hours.

Tue, Aug 20, 2002 Anonymous Anonymous

I want to extend my congratulations to Alan and his team for pulling it off. Can you imagine if it would have been 5000 users instead of 25. Microsoft claims that they are the best programmers in the world and if that claim is true, then why are the guys in the trenches banging their heads against the wall. M$, give us the tools to make our jobs a little bit easier.

Wed, Aug 7, 2002 Louis Atlanta

Wow I should have had a V8 and we should have migrated instead of running an upgrade. This articles had some great information. But having a backup Exchange server ready to go online saved my caveat!!

Sun, Aug 4, 2002 Anonymous Anonymous

excellent practical experience and superb links for preparing the upgrade. More such articles.

Thu, Jul 25, 2002 Anonymous Anonymous

It's great to read articles from someone who actually works in the trenches and gets their hands dirty. This article does an excellent job of overwhelming you with very necessary information that could help eliminate a lot of headaches.

Tue, Jul 23, 2002 David Perdue Great Falls, MT

Oh my goodness. This is just what I've been looking for in an article about an Exchange Migration. Great links. Good level of detail. As well as all of the nervousness and apprehension involved in a taks like this.

Mon, Jul 22, 2002 Dot Harris Chicago

Lab testing is par these days, but in actual implementation there is always some unknown that can't be foreseen or anticipated which is why an upgrade/installation window should allow for a backout plan. Great article.

Fri, Jul 19, 2002 Anonymous Anonymous

In response to Philip from Vancouver...
I have seen more "jerks" with degrees who think they know it all and who think a degree gives them all the tools to do the actual work make more mistakes than those who learned from experience. Just to enlighten you-- a degree is not always the answer and does not guarantee anything. It does not keep someone from being a jerk for sure. I suspect that I would much rather have someone with Alan's attitude work for me than someone like you.

Thu, Jul 18, 2002 Phillip Vancouver, BC

Eh. No offense but it sounds like you got cocky and did a half assed job
preparing. Admittadly you were working with a pretty poor set of tools
to begin with (Exchange) but as an "engineer" you should know these
limitations and fully prepare for them ahead of time. For such
an important project you should've taken steps to verify the backups, hell
I'd have cloned the original setup and mail database on a test machine and
performed the upgrade in a dry run before tinkering with the
customer network, but that's me.

So where did you get your engineering degree? Or are you one of those guys
that thinks a few years of helpdesk work and your big bad MCSE makes you a
real engineer? Thought so.

Thu, Jul 18, 2002 Dave Anonymous

Been there done that. Some pain still lingers. Going to do guidgen this weekend to (hopefully) end the pain. Migrate, do not upgrade. Check those KB's and learn the AD connector. As stated, do your homework. If you work through this type of pain your a better engineer for it. That's the bottom line.

Thu, Jul 18, 2002 Ellis McVea Tacoma WA

I resemble Alan's pain in the Exchange upgrade. Went through all he did and a few more agonies. Good article.

Thu, Jul 18, 2002 Anonymous Anonymous

Very well written article! Thanks.

Thu, Jul 18, 2002 Cisco User Fremont

Agreed-you should always migrate rather than upgrade anytime you can. Upgrades are almost always associated with problems.

Thu, Jul 18, 2002 endo oslo, norway

Great article ! There are so many things that can go wrond during a operation like this so i appreciate the tip of not upgrading but to migrate.

Wed, Jul 17, 2002 Anonymous Anonymous

Clearly written. Appreciate the inclusion of reference KB articles.

Wed, Jul 17, 2002 Pete Los Angeles

nice general overview of some caveats and failures that can occur

Wed, Jul 17, 2002 Anonymous Anonymous

Great information delivered in an easy to understand fashion!

Wed, Jul 17, 2002 Anonymous Anonymous

Great article! realy liked the inclusion of KB's and all the bloody details.

Thanks!

Wed, Jul 17, 2002 Anonymous Anonymous

Great article! realy liked the inclusion of KB's and all the bloody details.

Thanks!

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Comment:
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.