Introduction
What do a former NSA hacker, a former defense contractor, and an expert in microcontroller hardware all have in common?
They now all work here at Cylance, and they are among a number of security experts we asked to weigh in on the still-unfolding, bombshell news article first reported by Bloomberg in early October.
For the unaware, Bloomberg’s cover story, The Big Hack, alleged the existence of a Chinese government espionage operation which sought to compromise the supply chain of a motherboard manufacturer called Supermicro by inserting microchips into them that would allow them to spy on American technology giants, including Apple and Amazon.
If true, the story holds enormous consequences for U.S. national security.
Even if not true, there have already been significant implications for Supermicro. The company has already lost 40% of their stock value at market open the morning the story ran – a real world, tangible consequence to what might be thought of as an otherwise “niche” cyber story.
Suffice to say, Bloomberg’s story has stirred quite a bit of controversy.
The story has already drawn swift, detailed, and forcefully-worded responses from Amazon, Apple, and others, which were then echoed by official statements made by the U.S. Department of Homeland Security and its British partner, the National Cyber Security Center. Even former White House Cyber Coordinator Rob Joyce cast doubt on it publicly.
But Bloomberg has stood by its story, and even published a follow-on story with more detail regarding impact in the U.S. telecommunications sector.
China’s official response (as quoted in Bloomberg) was intriguing in that it suggested that they were victims of supply chain compromise too, with a wink and a nudge at the U.S. government.
On Monday of this week, news reports indicated that the CEO of Amazon’s Web Services was calling for Bloomberg to retract its story, something Apple’s CEO had done in an interview with Buzzfeed at the end of last week.
Cylance Experts Weigh In
Regardless of where the truth of any of this ultimately lies, the story and the responses to it have raised interesting questions about hardware and supply chain security.
And they have thrust technical details about rather arcane aspects of those disciplines into the limelight.
Readers without a technical background may be wondering what it all means. So, Cylance put a few casual questions to our own experts for their reaction.
Here’s a look at a conversation that began around the watercooler at Cylance.
First, let’s introduce the Cylancers:
- Jeff Tang, former NSA global network exploitation and vulnerability analyst and Cylance Senior Manager of Applied Research
- Scott Scheferman, former U.S. defense contractor and Cylance Senior Director of Global Consulting Services
- Erik Walthinsen, expert in microcontroller hardware and Cylance “Red Team” Professional Services Consultant.
Now, here some of their personal reactions and opinions.
Q & A
What was your overall reaction to the Bloomberg story?
TANG: I was pretty incredulous at the initial Bloomberg story. This is BadBIOS all over again. Every little detail can be plausible, but the overall story is improbable. Placing the blame on China is extremely convenient in a classic red scare/red herring situation.
WALTHINSEN: I thought it sounded plausible, but the details smelled wrong. It almost seemed like Bloomberg got punked (by China?), maybe with the intent of creating a big fat red herring that can be torn down and used as FUD [Fear, Uncertainty, Doubt] later when there really was a physical implant found?
SCHEFERMAN: My first reaction was, “Holy hell, this might partially explain the ‘Chinese abatement of malware activity’ post Obama-Xi handshake…. they just moved to hardware and stopped innovating/using software (malware) TTPs.” But that reaction quickly faded once all the skepticism and undercurrents started to get exposed. At that point this became either the biggest bombshell ever in the cyber realm, or the most egregious reporting ever on such an important national security topic.
What was believable? What, if anything, strained credulity?
TANG: Each individual facet of the story is believable, but combined, they paint an overly complex Rube-Goldberg contraption worthy of being the successor to Die Hard 4 (cyber fire sale). The reality is that China doesn’t care about OPSEC [operational security] when it comes to cyberoperations. China doesn’t care to be stealth. Investing so much time and effort to fabricate a rice grain chip and manipulating order volumes so manufacturing gets subcontracted out so planted PLA operatives can install the chips is unbelievable when taken as a whole.
The second [Bloomberg] story read like a massive spy thriller where the denouement is a product advertisement for Sepio Systems. A brief look at their website doesn’t uncover any magical unicorn technology. They are watching the network and cataloging the devices that show up. Talking about power consumption is like trying to distract the audience.
The lowest levels of network traffic don’t include power consumption. This is either sloppy writing or bunk. Any anomaly in the power consumption doesn’t magically get transmitted over the Internet. The Internet is a digital communications system which speaks over established protocols; power variances don’t get transmitted outside the network.
WALTHINSEN: There are definitely software flaws in the Supermicro BMC that pretty much give full access; this is a known problem. It's also believable that the Chinese manufacturing chain has been compromised by the Chinese government with the intent of covertly altering designs and inserting some kind of "spy" device.
However, the claims made in the article start to fall down with the main picture, showing the supposed device they retrieved. First, the part is clearly factory-fresh straight from the component tape, rather than destructively extracted from inside a PCB. Second, anybody with any experience messing with modern radio chips (e.g. Wi-Fi/BlueTooth) can identify that part from a mile away. It's an RF balun, used to convert the differential signal from the radio chip into a single-ended signal to connect to the antenna.
From an engineering standpoint, it's possible that a 6-pin device such as that could be used to accomplish something. The question is, what, exactly? On its own, lacking any "hack" support from anywhere else on the board, there's really no chance it could do anything. Of the six pins, you need two for power and ground, and the remaining four pins aren't much use.
In theory, they could be connected to the side of the SPI bus that connects the ASpeed-supplied BMC chip with the flash memory chip that holds its firmware. It could potentially do something nefarious to the firmware as it's loaded into the BMC, but there are a significant number of hurdles there.
First, to alter the data it really needs to sit between the two chips, not to the side of the bus. That would require several additional pins. Second, the volume of silicon that is represented by the tiny black dot in the middle of the pictured part would be very hard-pressed to make any useful changes to the firmware, especially since the BMC firmware can be updated. Instead of being able to alter a specific portion of the firmware, it would have to be smart enough to find what to change, and then change it. This would require even more implausible operations, as it would have to read the firmware independently from the BMC chip just to find what to alter.
It's also possible that the chip acts as an I2C bridge, as that is the protocol used to communicate some of the operations between the BMC and the rest of the system. I2C only needs two pins, so the four non-power pins would be enough to "break" the bus and operate as a proxy. However, the kinds of operations that occur over that bus are significantly more limited than would be required for the kind of "hacks" that the article references.
SCHEFERMAN: The fact that there were more than a dozen sources and the article took over a year to write initially provided credibility. The fact that it was Bloomberg publishing it did, too. But then we quickly learned that several sources were refuting the story’s accuracy. Meanwhile, the rest of the security community began to express doubts in terms of the technical feasibility of what was reported. A proliferation of potential motives behind the story began to accumulate in public discussions, including: trade war, short sale/manipulation, even the possibility of Supermicro’s competitors being involved – all rumors and speculation that added to the noise and confusion surrounding the story. And while this noise was loud, it did not directly impact the story’s credibility – though it did appear to focus critics of various stripes into the same general consensus of, “Show us a real spy chip on a motherboard in the wild, and have it vetted by a trusted third party.”
What was your reaction to the formal responses from Amazon, Apple, et al?
TANG: The formal responses from Amazon and Apple contain shockingly strong denials. They don’t have the air of vagueness we typically see in company responses. Typical denials take the form of: “There is no evidence of X”, but responses in this case are just: “it never happened”.
WALTHINSEN: I wasn’t really surprised about the denial, as you would expect to see something like that. I was more surprised by the forcefulness of the denial letter sent to Congress. If there really was something covert going on, the language of that letter would be a lot softer and more circumspect.
SCHEFERMAN: Initial reaction was that the responses were atypically ‘hard line’ and explicit. Either they had high confidence in their statements, or else the tough language was designed to cover up something big. I now believe the statements are earnest statements, but I also am not confident that they have been able to rule out the existence of supply chain implants of some sort in their environments… It’s not a leap to think there are/were implants in either organization, even if it is simply vis-à-vis malicious firmware already present.
What aspects of the technical descriptions offered by sources in the Bloomberg story rang true to you? Which did not?
TANG:
- Hardware implants exist – Fact
- Supply chains are vulnerable – Fact
- Tiny chips exist – Fact
- Firmware backdoors exist – Fact
- Firmware malware can subvert operating systems – Fact
- Ethernet ports have plastic shielding instead of metal – Myth
- Removing the implant chip without performing additional investigation – Myth
- Removing the implant chip and continuing to use the same hardware – Myth
WALTHINSEN: A big red flag for me was the Supermicro denial, which I found to be actually rather squirrelly from a technical standpoint. It specifically talked about how Supermicro "doesn’t design or manufacture networking chips or the associated firmware.” This tells me that they know what's really going on, and it has to do with the BMC firmware supplied by Aspeed.
It seems to me like Bloomberg got a lot of sources telling them what could happen, and spun a lot of the conjecture into what they claim was actually happening with Supermicro.
The reality is that it is absolutely possible that something like this is going on in the way they describe, but I strongly doubt that the specifics of this scenario are even remotely accurate. Because of the way boards are procured, well-placed agents of the PRC would be able to intercept the designs for motherboards such as this, and potentially insert some kind of chip inside the PCB.
The first problem with that is that the techniques required to insert components inside the PCB itself are quite new, though the kinds of assembly houses that could do it would likely overlap well with the houses that (e.g.) Supermicro would contract for those boards. The physical scale required is more along the lines of an iPhone motherboard, while server boards are much bigger.
However, the "chip" they would implant wouldn't look anything like the part pictured in the story. For one thing, the pictured part is almost as thick as the entire PCB, and thus impossible to embed given the number of layers in such a board. The central core of that board is probably less than half a millimeter thick, with the rest of the typically 1.6mm thickness taken up by 6 to 10 layers of copper signals and the intervening fiberglass. A real implant would be more like a "chip-scale package" where the silicon die is almost entirely on its own, rather than packaged in ceramic. Such a chip could definitely be placed in a carved-out well in one of the inner layers.
To connect it into the system, the number of pins depends on what all you're trying to accomplish and how. The artwork for the board would have to be altered in order to route the required signals on the inner layers to the chip in question, but the external layers have to be unmodified in order to pass a visual inspection. Given how many times these signals typically traverse from one layer to another, that's very likely possible in most cases.
There's also the inspection angle - such a chip should be relatively easy to detect with the proper X-ray equipment. A standard part of fabrication on board like this is X-ray inspection of the various components that are soldered down in such a manner that they cannot be visually inspected. If you don't see metal legs coming out of the sides of a part, it was likely X-ray inspected. For server boards like that, I would say it's likely that they do automated inspection of every such chip on every single board, as the time/cost of that is going to be far less than having a failed board work through the supply chain. While these are typically fully automated, it's normal for a sampling of the images to be sent to the customer for validation. Thus, the assembly house is going to have to run enough unmodified boards in order to have a decent number of "faked" images to send. If the Chinese government is behind it, that's not much of a barrier.
Now, in a second article Bloomberg’s sources claimed that devices were found inside the Ethernet connectors. The claims made in this article are, in my opinion, outright laughable, honestly. The assertion is made that it's the "metal" Ethernet connectors that are the problem, compared to the older plastic housings. They assert that the metal housing is there to "dissipate the heat of the chip," when in fact it's there to shield the electromagnetic emissions of the "magnetics" inside the connector. These are called "MagJacks" and were invented years ago to save board space, as the transformers used to be in their own packages and thus took up a lot more room.
From a signals perspective, there's also the minor detail that it's effectively impossible to sniff any Ethernet signals from inside the jack, on any modern device. An older device that runs at 10 or 100Mbps (a.k.a. before Gigabit was first on the market in 2000) will transmit their signal on a single pair of wires (out of the four pairs in the cable) and receive from the other end on another pair. There’s nothing particularly funky going on in that transmission, and anybody passively observing the signal can trivially determine what the data are.
However, in a Gigabit connection, the signals are overlapped using what's called a "hybrid." This is an ancient technology that literally dates back to the days of Thomas Edison’s, "Mr. Watson, come here," as every landline phone uses a hybrid. What it does is put both signals (from A to B and B to A) both on the same wire pair at the same time. The only way this works is because for each unit, it can "subtract" the transmitted data from the signal on the wire in order to retrieve the signal from the other end. This is only possible because it is the transmitter.
Consider a landline phone with two auctioneers going full tilt. If you were to listen to the phone line itself, you'd hear both auctioneers simultaneously, and make no sense of it whatsoever. However, if you are one of the auctioneers, while you're hearing the muddled signal of both talking, your brain is designed to remove your own voice from the mix before trying to make sense of it. But just like with the Ethernet chips involved, that subtraction is occurring entirely within your head.
In the case of a MagJack, the combined signal is all it ever sees. It has no idea what the Ethernet chip it's connected to is actually sending, thus it cannot separate the two signals.
So, fundamentally, the idea of a MagJack being host of a spy chip is outright lunacy.
SCHEFERMAN: The second Bloomberg article cited a source claiming that ethernet ports with metal surrounds are an IOC [indicator of compromise] that the port may have an implanted chip (rationale being that the metal acts to dissipate heat from the added chip's workload):
" [...] one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer."
This was the moment that, taken together, both Bloomberg articles had jumped the shark. Surely the authors could have Google-searched for "ethernet port on motherboard" and found dozens of photos of motherboards with ethernet ports surrounded in anti-EMI metal? How is it that while the first Bloomberg article took many months to write, this second one only took a few days to come out after this source called them up to share such un-vetted insight?
Conclusions?
As you can see, there are areas of agreement among our experts, as well as areas of disagreement. That just goes to show that it may be a long time before any hard and fast conclusions about The Big Hack can be drawn.
At the end of last week, the head of the U.S. intelligence community, Director of National Intelligence Dan Coats, said in a speech that while he’s seen no evidence to support the claims made in the Bloomberg stories, the intel community is “always watching.”
“Be aware of supply chain threats,” he was reported as saying, before urging his listeners to realize that supply chain threats are an “insidious problem.”
That last point, about remaining vigilant to potential threats to the supply chain, is sound advice and something all our experts – and readers – can agree on.
EDITOR’S NOTE: This blog is for informational and entertainment purposes only. The views, thoughts and opinions expressed in this blog post represent the opinions of the authors/named parties only, and do not represent the views or opinions of Cylance, Inc. or its partners or affiliates. Any and all liability for the content of this blog post or any omissions, including any inaccuracies, errors, or misstatements in such data or information is expressly disclaimed.
