In-Depth

Breakdown of the Rackspace Ransomware Incident

What caused the issue and what, as customers, we can do to keep our data secure.

• By Joey D'Antoni
• 12/15/2022

On Dec. 6 Rackspace confirmed what was widely reported in tech media and on various social media platforms: Its hosted Microsoft Exchange platforms had been hit by a ransomware attack. Since then, based on reports on Reddit and Rackspace's own twitter accounts, customers have been able to start using Microsoft 365 email again, but Rackspace hasn't helped customers restore any email from before the outage.

In an interview with Barron's last Friday, Rackspace reported that tens of thousands of small and medium-sized companies were affected by the outage. Based on the customers I spoke to in researching this article, Rackspace's communication to their customers has been extremely limited.

Rackspace announced the outage at 2:49 p.m. EST on Dec. 2, and moved slowly to declare this a ransomware event. The company then acknowledged a "security incident" on Saturday, Dec. 3, specific to their hosted Exchange environment, requiring Rackspace to take servers offline. In its first announcement of the security incident, Rackspace acknowledged a fix could take "several days."

I spoke with Rackspace's chief product officer, Josh Prewitt, over a Zoom call for this article after Rackspace PR reached out to me. Prewitt acknowledged that this attack fully affected the company's hosted Exchange service offering. Rackspace is working with CrowdStrike to research and identity the details of the attack. The investigation is still ongoing, but CrowdStrike has been able to confirm that the surface area of the attack was only the hosted Exchange environment and not any of Rackspace's other services.

Prewitt also said that 75 percent of Rackspace customers can send and receive email, which shows they have successfully migrated to other platforms like Microsoft 365. However, that migration only includes customer certificate configurations so they can use Microsoft 365, and does not include existing email and calendars stored on powered-down servers in Rackspace. Prewitt mentioned that a sizable percentage of Rackspace customers did have the optional archive service, which allowed them to quickly download their emails. He also acknowledged Rackspace is working on a solution to allow customers to be able to restore their archives (but that is currently a work in progress).

I asked Prewitt if there was any way to recover this environment from a backup or a snapshot. He replied, "The way the environment is architected is it takes advantage of the native clustering that's built into Exchange. We've got multiple copies of everything, and Exchange is going to naturally distribute that out to other servers within the cluster. And so everything would have had at least three copies, depending on the datacenter that was in."

He added, "When we talk about the ransomware incident and how quickly we had to respond to being able to make sure that we contained it, we shut down everything. We shut down power and network for the entire environment. And that includes the backup environments there, as well, too. We've been working towards how we give customers access to the data so that they can download an archive."

Rackspace has said in its filing with the SEC that it maintains cybersecurity insurance commensurate with the size of the business and is confident in its ability to absorb potential financial costs associated with the ransomware incident. According to customers I spoke to, Rackspace is offering relief through migration assistance. However that assistance does not fully cover the time and effort customers are putting in to try to get their business back online.

While downtime and ransomware attacks can happen, cloud providers should have higher standards than the average IT organization. Cloud providers have a couple of roles to play here. First, they must communicate to their customers openly and quickly. As much as I love database systems, I always acknowledge that email is the most important business system, especially for smaller organizations that may have limited other channels of communications. I think Rackspace erred on the side of under-communicating until they fully understood the problem, which is understandable but leaves customers in a challenging situation.

Second, and probably more importantly, a cloud provider like Rackspace, Amazon Web Services or Microsoft Azure services multiple customers, so they need to do their best to ensure they are following security best practices and basic principles of isolation.

While I don't have direct knowledge of the architecture of Rackspace's environment, I can offer a couple of insights. Given that this attack affected all of Rackspace's hosted Exchange customers, we can make some basic assumptions about the architecture within. It is likely that Rackspace was using Exchange multi-tenancy, which means different customers run on shared hardware. Also, because the attack affected multiple datacenters, there was likely limited network isolation for this environment. It seems unlikely that administrative accounts were configured with advanced security features like just-in-time roles (that would limit lateral movement in the environment).

One of the key concepts in cloud computing and site reliability engineering is a "blast radius." This means reducing the impact of any hardware or software failures from cascading and impacting other customers. Having an attack that can take down an entire service for two weeks and counting is simply unacceptable as a cloud provider.

What About Microsoft 365 or Gmail?
If your organization is using Microsoft 365 or Google Workspace, you have a bit more protection. Microsoft and Google use a single-tenant model, which gives added isolation, reducing the risk of another organization's problem becoming yours. These providers are the best at managing what they do, and they both "own the code" for their respective services, which allows them to be more flexible in how they patch and support those services. Both services also supply some level of redundancy across cloud regions, allowing them to supply higher uptime and provide granular and robust security controls for both administrators and users.

However, as much as I like the Microsoft and Google services, you need to protect your data within those services. While both providers deliver redundancy and high availability, you can and should take backups of that data to other sources. There are many third-party offerings from seemingly every backup solutions provider for both Google Workspace and Microsoft 365. I can't recommend backing up this data highly enough -- beyond just cloud failure, it can also protect you from accidental deletion of data by users. While you should expect a higher level of data protection from cloud providers, you still need to ensure that you have your data in case things go terribly sideways.