Posey's Tips & Tricks

Coping with a Battery Backup Failure

Brien discusses his process of replacing a backup battery and gives some insight on how to avoid this issue in the future.

Even though my organization is essentially a one-man operation, I have always tried to take advantage of any available safeguards to keep my systems up and running and to keep my data protected. One of the safeguards that I use is battery backups for every device on my network.

A few days ago, one of my backup batteries failed. The failed battery made an obnoxious beeping noise that I couldn't mute, so I immediately went to the store to get a replacement battery. Upon my return, I checked to see which battery backup had failed. It turned out to be the only battery on my network that was supporting two devices -- my network switch and my wireless access point. This is where things got interesting.

Even though the old battery backup had failed, the devices that were plugged into the battery were still receiving power. Unfortunately, every device on my entire network was connected to either the switch or to the wireless access point that was plugged into the failed battery. Hence, replacing the battery would mean losing connectivity across my entire network.

In a way I was lucky because I work out of my home and nobody uses the network except for my wife and I. Even so, various logistical factors related to network usage made it impossible to replace the battery for three days (that was three days of the old battery beeping constantly).

Before I swapped the battery, I took a few precautions. First, I paused the replication process on some Hyper-V servers so that the loss of connectivity would not result in replication errors. I also have a Hyper-V cluster that I knew would lose quorum as soon as the network switch went offline, so I took the cluster offline and shut down the nodes.

It took me less than an hour to take everything offline, replace the battery and bring everything back up. Even so, the whole event left me thinking about what I might do differently in the future so that I might avoid the type of situation that I just described.

In large, this type of failure in enterprise-class organizations is typically a non-issue. Enterprise class networking hardware often contains redundant power supplies. This means that a single switch could theoretically be plugged into two separate batteries. That way, if a battery were to fail then it could be replaced without having to take the switch offline.

The problem with this approach is that I work out of my home. Even though most of the hardware that I use is much higher end than what the typical home user would purchase, using enterprise-class hardware is overkill. It's hard to justify the cost of enterprise-class hardware when you only have two users on your network. This made me stop and think about what a small- to mid-sized business might do to prevent the type of situation that I described earlier.

Rather than using high-end hardware with redundant power supplies, a smaller organization could effectively achieve the same end result by using redundant switches (each plugged into a separate battery). Of course for this concept to work, each network device would need a minimum of two connections -- one to each switch. That way, if a switch failed (or had to be taken offline) there would still be a path that network packets could use to reach their destination.

Windows Server 2012 can actually help out a lot with making this type of design work. I have seen a couple of examples in the past of redundant switched networks. The problem with such networks is that getting network devices to use an alternate network path when a failure occurs is sometimes tricky. For whatever reason, some devices try to continue using the old network path, even though an alternate path is available.

With Windows Server 2012, you can use the NIC teaming feature to make a network switchover seamless when a failure occurs. A NIC team can be designed to include a standby NIC. A standby NIC is a NIC that remains unused unless a hardware failure occurs. This means that you could connect a standby NIC on each server to a switch whose only job it is to provide an alternate communications path in the event that the primary switch goes offline. The nice thing about this approach is that if configured correctly, you can get away with using commodity network hardware.

As you can see, something as simple as a battery backup failure can have a major impact on your network. It's tough to do anything about the disruptive nature of such a failure after it happens, but with a little advanced planning, you can completely avoid having a major outage while you replace a battery.

About the Author

Brien Posey is a seven time Microsoft MVP with over two decades of IT experience. As a freelance writer, Posey has written many thousands of articles and written or contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and healthcare facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. When He isn't busy writing, Brien Posey enjoys exotic travel, scuba diving, and racing his Cigarette boat. You can visit his personal Web site at: www.brienposey.com.

comments powered by Disqus

Reader Comments:

Tue, Mar 26, 2013

Another way to avoid the problem is to know that UPS batteries are designed with a max useful life of three (3) years if running in an environment that is around 70 degrees. If the UPS is subjected to temperatures of 90/95 degrees, that 3 years is reduced to 1.5 years. By replacing batteries in advance of their dropping dead, you avoid the problem altogether.

Mon, Mar 25, 2013

It's a little early for April Fools jokes . . . you might want to recover by claiming this was an article that was to come out next Monday. Sorry.

Mon, Mar 25, 2013

I think Mr. Posey is messing with us. I can't fathom a MVP with 20 yrs. isn't aware of hot swappable UPS batteries. I picked up on the pitch for Server 2012. But I thought this article was suppose to provide something useful. That's just wrong.

Mon, Mar 25, 2013 Ron Coral Springs, FL

It doesn't take an IT person to figure out the pitch was for windows Server 2012, don't get me wrong I run Server 2012 as well as every other version of Windows server. Regardless of it being a server or a switch I have swapped out UPS batteries while the gear is live, instead of shutting anything down. I think you should come up with better examples, or you won't have any readers left. Actually the last time I remember having to bring gear down to change a battery was a 1KV Topaz in which you had to take a piar of pliers to disconnect the battery. I remeber this because one time I arc'd the battery terminal with the board, unplugged and turned off ant it melted a hole in my pliers. (I still have that pair as a reminder what a UPS can do even unplugged.

Mon, Mar 25, 2013

I've not worked with a bypass switch. However, if you're just replacing a battery, you should have a UPS that allows hot swapping the battery. APC Smart-UPS allow this. It only takes just a few minutes. Am I missing something with this article?

Mon, Mar 25, 2013

You should invest in an external maintenance bypass switch similar to a micropod. The switch allows yoo to remove the UPS for service while maintaining the load on commercial power. Only risk is the loss of commercial AC while the UPS is off-line.

Mon, Mar 25, 2013

There are Many UPS's sold even in Smaller sizes Like 1kva and 1.4Kva that you a user can change the Battery with out ever having to power off the UPS they have quick connectors I have done it several time in my office when battery's have failed and you just hope there is no power outage while you have the batterys disconnected.

Mon, Mar 25, 2013 IT247

Our battery backups,which were not expensive ones, lets us slide the batteries out and swap them while everything stays running on electricity.

Mon, Mar 25, 2013 Guy

That's what maintenance bypass switches are for.

Mon, Mar 25, 2013 Guy

That's what UPS maintenance bypass switches are for.

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Comment:
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.