Datacenter Trends

Chinese Spy Chip Scandal: Advice for Datacenter Managers

• By John K. Waters
• 10/09/2018

A Bloomberg Businessweek report alleging that a Chinese manufacturing subcontractor installed tiny spy chips on motherboards in servers used by Amazon Web Services (AWS), Apple, the U.S. government and about 30 other organizations went off like a bomb when it was published last week.

Apple and AWS issued strong denials. So did San Jose, Calif.-based hardware maker Super Micro Computer (also known as SuperMicro), in whose servers the chips were reportedly installed.

"The security of our customers and the integrity of our products are core to our business and our company values," Super Micro said in its statement. "We take care to secure the integrity of our products throughout the manufacturing process, and supply chain security is an important topic of discussion for our industry. We still have no knowledge of any unauthorized components and have not been informed by any customer that such components have been found. We are dismayed that Bloomberg would give us only limited information, no documentation, and half a day to respond to these new allegations."

This week, the U.S. Department of Homeland Security weighed in, stating, "At this time we have no reason to doubt the statements from the companies named in the story."

So far, Bloomberg is sticking with its reporting.

True or not, the report raises an obvious question: What can datacenter managers do to ensure the security of their organizations' hardware? The short answer is: nothing simple, easy or cheap.

Moreover, says Joseph Fitzpatrick, an instructor and researcher at SecuringHardware.com who was interviewed for the Bloomberg story, there's probably nothing you need to do.

In fact, Fitzpatrick expressed serious misgivings about that story this week on a Risky.Biz podcast, during which he said the hardware back-dooring Bloomberg described "didn't make sense."

"Hardware implants are real and technically possible," Fitzpatrick wrote in a blog post. "We see modchips, counterfeit bypass devices, keystroke loggers, and card skimmers all the time, but we've never actually seen deployed hardware backdoors in servers."

I contacted Fitzpatrick to get his advice for datacenter managers, and he pointed me to a long list of recommendations he offered in that blog post. "It's a bit harsh," he said in an e-mail, "but it's the message that needs to be said."

It's a great post -- blunt, but not actually harsh -- and a must-read for anyone who wants to get a specialist's un-mediated take on this issue. Fitzpatrick has spent more than a decade working on low-level silicon debug, security validation and penetration testing of CPUs, SoCs and microcontrollers. He also teaches classes on applied physical attacks.

Fitzpatrick offered a number of observations and recommendations in his post. I think these are his top three:

• It's unlikely you're affected. Really. Even assuming every claim is true, and even if there is a secret device on every single X brand motherboard, it's unlikely you're targeted by whatever payload the implant carries.
• There are no published indicators of compromise (IOCs). The device and placement referenced in the article are only representative and not actual devices. Having experienced hardware eyes on your board might pick out something odd, but won't be conclusive.
• Without an IOC, you need to do a time consuming, thorough, invasive, destructive analysis of every component on your board. This is expensive [emphasis his]. If it's not time consuming, invasive, destructive, and expensive, you're not getting a thorough job.

And what if you do find something?

"Take off your tinfoil hat," Fitzpatrick wrote. "If something is different, rule out an engineering or business reason before assuming malicious intent. Sometimes docs are outdated, chips are interchangeable, board errors are fixed, and market prices for compatible replacements shift. Next, rule out simple profit motives. Bunnie [Andrew "bunnie" Huang, famous hacker] has an excellent coverage of the range of counterfeit devices. Pretty much every picture you've ever seen of a hardware implant is really a counterfeit bypass device. If you've gotten this far, you're on to something. If you're bold, you could be the first to publicly disclose full details of an actual malicious hardware implant. Double up on your foil, find a bunker, use tor, use signal, and tell everyone."

There's lots more in Fitzpatrick's post, which you can read in full here.