Posey's Tips & Tricks
Critical Considerations for Server GPUs
Server GPUs offer powerful performance for AI workloads, but IT pros must weigh critical factors -- like form factor, power requirements and workload compatibility -- before installation.
Although I have been running various AI workloads on desktop computers for a while now, I recently made the decision to try running a particular workload on a dedicated server. This workload requires far more memory and processing power than what a desktop computer can deliver, so running it on server hardware was the only practical option.
Before I could begin hosting this workload on server hardware, I needed to install a GPU into one of my servers. While this step would probably seem simple enough (and it is), there were actually several things that I had to consider before getting started. I wanted to share with you some of these important considerations.
The first of these considerations was the GPU's physical size. Yes, the GPU needed to fit into a PCIe slot, but that isn't what I am talking about. The server that I originally wanted to install the GPU into adhered to a 1U form factor, meaning that there is a limited amount of space within the chassis. Many modern GPUs are twice or perhaps three times the thickness of the average PCIe card, making it impossible to install them into a 1U server.
It isn't just the GPU's thickness that matters, but also its length and height. The PCIe slot shown in Figure 1 for example, is relatively small and can only accommodate low profile expansion cards. However, my server has a second expansion slot that can accommodate a longer card and with a full height bracket.
[Click on image for larger view.] Figure 1. The case design prevents the use of oversized PCIe cards.
Another consideration to take into account prior to purchasing a GPU is whether your server requires a riser in order to accommodate PCIe cards. As previously mentioned, I had originally intended to use a 1U server. Because a 1U server is only a couple of inches thick, the case design prevents PCIe cards from being installed vertically as they would be in a PC. Instead, a riser plugs into the system board and then the PCIe card mounts within the riser. The reason why I am mentioning this is because it is very common for low profile servers to require the use of a riser, so if your server does require a riser then it is important to make sure that you actually have the riser in your possession before attempting to install a GPU.
This brings up another important point. Low profile servers like the 1U model that I am using are normally very limited with regard to the number of PCIe devices that they can accommodate. As such, it's a good idea to check to make sure that your server has a slot available for use.
As an example, my server only has two PCIe slots. One of those slots is being used by a 10 Gbe card that I use to connect the server to a storage array. The other slot was occupied by a standard gigabit NIC. That being the case, I had to remove the network adapter, as shown in Figure 2, and use the server's integrated network ports instead.
[Click on image for larger view.] Figure 2. I had to remove a network adapter to make room for the GPU.
Yet another thing to consider prior to trying to install a GPU into a server is the GPU's power requirements. Early on, before I came to my senses, I thought that perhaps I could avoid having to pay for a server grade GPU by using a spare Nvidia RTX 4090 that I had laying around. The 4090's physical size is way too large for the GPU to be able to fit inside a 1U server. Even if that weren't the case however, there is another problem. The PCIe slot cannot provide enough power for the GPU by itself. The Nvidia RTX 4090 must receive supplementary power through a direct connection to a PC's power supply. This is not unique to the 4090. Many larger GPUs require supplementary power and server's often lack the required power cables.
This is not to say that you can't connect a power cable to a GPU. I ended up installing an Nvidia A4000 GPU into my server. This card requires a six-pin power adapter. The server that I am using has a mini 8 pin power connection on the system board. I was able to acquire a special cable that allowed me to power my GPU from this connection.
[Click on image for larger view.] Figure 3. This GPU does not require the use of power cables.
Finally, before purchasing a GPU for your server, it's important to consider how you will be using the GPU. Some GPUs are designed to service VDI deployments. These cards actually contain multiple GPUs, each of which can be allocated to a separate virtual desktop. Other GPUs are designed primarily for deep learning, while other GPUs are better suited to AI applications for which the training process has already been completed. As such, it's important to choose a GPU that is well aligned with your workload's requirements.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.