Never Again

Making Sure the Backup Backs Up

Testing and backup problems result in an all-nighter for a health care IT staff.

It was two days after what appeared to be a run-of-the-mill migration from a Windows 2000 File Server and PDC to a shiny new Windows 2003 Virtualized File Server and domain controller (DC), when our Citrix farm decided to crash around 3 p.m. Because our entire medical practice relied on Citrix to deliver the electronic medical record system to our providers and staff, we gang-tackled the problem immediately.

Diagnosis: Cryptic
With the help of my supervisor we spent the next 16 hours trying to diagnose the problem. We had five Citrix boxes that were responsive, but Citrix was throwing very cryptic errors when trying to connect. As the night wore on we reinstalled Citrix on the licensed server and ran updates, even reinstalling it on the servers, thinking if we could get just one of them up we could worry about the others later.

We found some information in a few forums that pointed us to the problem being in the Web interface module of Citrix. So we created a new Web interface server around 4 a.m., with the clinic scheduled to open up again at 8 a.m.

Prescription: Update
After configuring the Web interface to point to an emergency server, we were able to succeed with our first connection around 7 a.m. The next most pressing problem was that all of the clients pointed to the old Web server. So for the next two hours we were answering calls and trying to update everyone's URL at once, because the version of Citrix we had didn't support automatic updating.

Consequently, we had to go to each computer and enter the URL for 75 increasingly impatient users. All of this was accomplished by 10 a.m. and we finally went home after 26 hours of long, continuous work.

Well, it turns out our original file server contained some special pointer for the Web interface, which had been installed by me and my supervisor's predecessors respectively, who no longer worked for the company. Once we switched the server the pointer was erased along with all the settings needed to connect to our Citrix app farm.

The moral of the story: Allow ample time to test, and make sure that the backup works.

About the Author

Chris Gahlsdorf is a systems administrator with Northwest Human Services.


  • Basic Authentication Extended to 2H 2021 for Exchange Online Users

    Microsoft is now planning to disable Basic Authentication use with its Exchange Online service sometime in the "second half of 2021," according to a Friday announcement.

  • Microsoft Offers Endpoint Configuration Manager Advice for Keeping Remote Clients Patched

    Microsoft this week offered advice for organizations using Microsoft Endpoint Configuration Manager with remote Windows systems that need to get patched, and it also announced Update 2002.

  • Azure Edge Zones Hit Preview

    Azure Edge Zones, a new edge computing technology from Microsoft designed to enable new scenarios for developers and partners, emerged as a preview release this week.

  • Microsoft Shifts 2020 Events To Be Online Only

    Microsoft is shifting its big events this year to be online only, including Ignite 2020.

comments powered by Disqus

Office 365 Watch

Sign up for our newsletter.

Terms and Privacy Policy consent

I agree to this site's Privacy Policy.