Posey's Tips & Tricks
Troubleshooting Windows Server iSCSI Connectivity Issues
When one of his Windows Servers stopped connecting to its storage array, Brien could find only an incomplete fix and an even weirder problem with Windows.
I recently ran into a weird situation regarding iSCSI connectivity in Windows Server. Even though it took me a while to fix the problem, I learned some cool tricks as I worked through the troubleshooting process -- but I also discovered a problem with Windows.
Before I show you what I learned, let me give you some background. My production environment is based around a pair of Windows Server 2019 servers that act as Hyper-V hosts. Both of these servers connect to external storage arrays by using an iSCSI connection across dedicated 10Gbps links. The setup has been in place for about a year and has been working flawlessly.
Recently, I decided to take some much-needed time off. A couple of days before I left on vacation, however, my home was struck by lightning. The previously mentioned servers were not damaged, but the lightning strike made me a bit paranoid, and I decided to power down and unplug the servers while I was gone.
Upon my return, I expected to be able to simply plug in the servers and power them up. After all, they were both healthy when I shut them down a couple of weeks earlier. However, one of the two servers had difficulty connecting to its storage array.
So how can you fix something like this? The first thing I recommend doing is opening the iSCSI Initiator Properties dialog box and seeing if Windows Server will find your target. To do so, go to the Targets tab, enter the target's IP address and press Enter. The Discovered Targets list should be populated with a list of the iSCSI targets that are accessible through the specified IP address, as shown in Figure 1.
Figure 1: The iSCSI Initiator Properties sheet should list your iSCSI targets.
If the targets are displayed, as they are in the screen capture above, then it proves that the network link between Windows Server and the storage array is working properly. If the targets are not found (and you have verified that you entered the correct IP address), the problem is most likely network-related. You might have a loose network cable or a network configuration error, or the storage array may be having some sort of issue. I have occasionally seen situations in which cycling a storage array's power corrected this type of situation.
In my case, Windows Server was able to discover the iSCSI target but was unable to connect to it. That meant that even though the network link between the server and the storage array was working, some other problem was preventing Windows from being able to connect.
When you select an iSCSI target and click the Connect button, you are taken to the screen shown in Figure 2.
Figure 2: You can normally connect by clicking OK.
Normally you don't have to do anything on this screen other than simply click OK (unless you need to enable multipath). Unfortunately, I just couldn't get the connection to work, so I tried clicking the Advanced button, which opens the Advanced Settings screen shown in Figure 3.
Figure 3: This is what the Advanced Settings screen looks like.
As I examined the Advanced Settings screen, everything appeared to be in order. The server was still configured to use a CHAP login, just as it always had been.
Not really knowing what else to try, I began experimenting with the drop-down menus at the top of the dialog box. Whet I found was that these drop-downs allow you to specify the adapter through which the iSCSI connection will be established, as well as the initiator and target IP. These options could be really useful if you want to force your iSCSI traffic to flow over a specific network adapter.
Even though none of the advanced options helped me directly, experimenting with the various settings did eventually lead me to the source of the problem. I noticed that when I made a change, the Apply button was never illuminated. If I simply made a change, clicked OK, then went back to the Advanced Settings screen, my change would be gone. This held true even after multiple reboots. In fact, I am writing this column two weeks after discovering the problem, and I still have not been able to save any of my settings.
So how did I end up fixing the problem? To be perfectly honest with you, I have not yet fixed the problem to my satisfaction. Realizing that the Advanced Settings dialog box was using default settings regardless of anything I tried, I theorized that maybe the dialog box was no longer recognizing my CHAP password. I temporarily disabled the password on my storage array, and the Windows iSCSI initiator connected without issue.
Obviously this is not a good long-term solution. My hope is that Microsoft recognizes the problem and releases a patch to correct it.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.