Intel's zero-day problem
In his talk at FOSDEM 2017, Georg Greve mentioned that every recent Intel CPU contains a second, internal CPU that you cannot audit but which can take over your machine. His contention was that this could be used to do bad things without your consent if it turned out to be treacherous or buggy. As of May 1, 2017, the latter prediction turned out to be worryingly prescient.
Out-of-band management is technology that allows managing a remote server, at all stages of its operation, without having to have an external IP-enabled keyboard, video, and mouse (KVM) device and a remotely operable power strip. It allows a server to be powered off and on, warm and cold rebooted, and gives remote access to the keyboard, video, and mouse once the system is far enough up that the graphics card has been initialized. Without technology like this, if a remote system can't pass its BIOS self-test, has a faulty GRUB configuration, or the kernel panics, someone has to get in a car, often in the middle of the night, and go to a data center to fix it.
In its early days, out-of-band management was accomplished by having a separate piece of hardware inside the server case, often with its own network port, which all had to be configured and connected in order to work. This photo of an IBM Remote Supervisor Adapter (RSA) II shows the VGA cable which has been looped around from the server's graphics card to the RSA card, so that the display can be re-digitized and made remotely available. All this extra hardware was expensive, so the feature tended to be available only on high-end servers. It often needed its own network interface card (NIC), which meant more cabling and switch ports, and more administration.
Intel's Active Management Technology (AMT) provides the same services without any external hardware overhead. It does this by piggybacking on top of the base system's connections, extracting its inputs from and sending its outputs to the system's NIC, usually on TCP ports 16992-16995, 623, and 664. AMT can read from and write to any piece of memory or storage on the machine, and read and write to the screen, without the host CPU being aware this is happening — and therefore, without the host CPU being able to log or audit any of it. Because of the network access, it can send anything it chooses anywhere it wants to, and receive inputs the same way.
Such capabilities are necessary for the AMT to do its job, and network access to the AMT requires authentication. But the AMT is provided by Intel's Management Engine (ME), the "second processor" to which Greve referred. It is present on all Intel's current chipsets, and most of what they've made in the past decade. The ME has total control over the entire system; it is closed and unaudited, and we have all historically been asked to take Intel's word that it is secure.
According to Intel, there is a vulnerability: the AMT can be given remote instructions without an authentication step. This is pretty awful, but it is perhaps not quite as awful as the media coverage is making it out to be. As Matthew Garrett notes:
The drivers that Garrett refers to are those that integrate the AMT and the operating system, allowing the system to pass requests to the ME. Intel calls this the Local Manageability Service (LMS). Fully mitigating the vulnerability requires disabling the LMS if it is installed.
A good first check is to see whether AMT is enabled on your system. To
start with, Intel said that the vulnerability "does not exist on Intel-based consumer PCs
"
For other systems, the company lists four
ways of finding out whether a system is AMT-capable; unfortunately,
two of them require rebooting the system and interacting with the console.
If the system is remote that is only possible if you've activated some
kind of out-of-band management solution in which case you already know
you have a problem. The other two mechanisms require you to be running Windows.
Garrett points out that a Linux system with AMT makes this information available
via lspci. Here's some output from one of my vulnerable
systems:
[tom@trouble ~]$ lspci|egrep -i 'mei|heci' 00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05) 00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)A non-vulnerable system would display nothing. Garrett goes on to say that even though a system may have AMT, that doesn't mean it's been provisioned, so the system still may not be vulnerable. Since my system does have AMT provisioned, I will need to reboot it, at which point I should be able to disable AMT via the BIOS. Once I've done that, the system will no longer be remotely manageable; for this particular system, that's fortunately not a problem, since I don't manage it remotely. That said, it's an important server and I can't reboot it during the working day, but it's behind a firewall and not accessible on arbitrary ports, so I should be reasonably safe until then. If it was a system that I needed to manage remotely, I'd simply be out of luck; as one of the people from a group (SemiAccurate) that claims it found this bug early—several years ago—said:
This vulnerability is definitely a real and extremely serious problem for people who've been setting up systems with internet-accessible out-of-band management capability and trusting Intel's assurances on access control to keep them safe. Those people have to badger their system vendors for patched firmware that incorporates Intel's changes. Cloud providers and other large-scale virtual-system vendors are probably having a really bad day, scrambling to patch their systems before the black hats get automated exploit tools up and running, which SemiAccurate strongly suspects has already happened. For those whose systems are from smaller vendors, or past end-of-life, or in the worst case self-built or white-box systems, there may be no fix available, and simply living without remote-management capability may be their only choice for the remainder of the lifetime of the system.
For the rest of us, this vulnerability is still a fairly big headache, but it probably isn't the end of the world. The deeper problem exposed by this is exactly the one to which Greve drew our attention: that every system we buy from a vendor who asks us to trust them that it all works exactly as advertised is a system that can cause us great pain if it turns out that the trust was misplaced. This time, we got off fairly lightly. But if the hole had been bigger, for example if a remotely-accessible way had been found to give arbitrary instructions to the ME, that really would look like the end of the world.
That might also be true for similar remotely-exploitable bugs in components like the CPU and the NIC, but we have some chance of catching them in action in the former, and the latter doesn't have anything like the range of access that the ME has. The ME is a black box at the very heart of the system, and that makes it a particularly awful device to be exploitable.
We dodged a bullet this time. If we have any
sense, we might insist our
vendors stop giving us their assurances that we're protected and start
giving us the details and access we need to establish it to our own
satisfaction.
Index entries for this article | |
---|---|
Security | Firmware |
Security | Hardware |
GuestArticles | Yates, Tom |
Posted May 3, 2017 20:07 UTC (Wed)
by mikemol (guest, #83507)
[Link] (27 responses)
I know there are people who've put their management ports on public IPs.
But still. *Jeeze*. Don't Do That!
Your management interfaces belong on private ports (or VLANs, if you must) on private broadcast domains reachable *only* via authenticated, secured VPN.
*All* code has bugs, so you hide away access to sensitive execution environments behind as many security layers as you reasonably can, to mitigate the impact in the eventuality that that one of those layers is flawed.
Posted May 3, 2017 20:56 UTC (Wed)
by sionescu (subscriber, #59410)
[Link] (22 responses)
Posted May 3, 2017 22:55 UTC (Wed)
by mikemol (guest, #83507)
[Link] (21 responses)
This was an inherent technical necessity with the masquerade-style NAT that's ubiquitous on IPv4, but it's by no means unique to it.
Posted May 3, 2017 23:06 UTC (Wed)
by raven667 (subscriber, #5198)
[Link] (7 responses)
Posted May 3, 2017 23:15 UTC (Wed)
by mikemol (guest, #83507)
[Link] (2 responses)
The firewall I described belongs on the network gateway. (In addition to the host.)
Posted May 5, 2017 15:55 UTC (Fri)
by raven667 (subscriber, #5198)
[Link] (1 responses)
Posted May 9, 2017 5:30 UTC (Tue)
by ringerc (subscriber, #3071)
[Link]
Posted May 9, 2017 5:26 UTC (Tue)
by ringerc (subscriber, #3071)
[Link] (3 responses)
Pathetically insecure devices that never get security audits or updates. That have no user or admin configurability or documentation. Devices hacked together by a rushed team adapting a related product's code in a single 36-hour sprint to try to beat a competitor to market, before completely forgetting about it.
Devices where if you're lucky you might be able to update or fix them with a vendor-pushed remote firmware update (assuming an attacker doesn't abuse the inevitably insecure mechanism instead). Or with a soldering iron to access pads for USB-serial, RS232, or even JTAG. Occasionally you'll have a webui to upload a firmware blob. But your chances of actually fixing anything without the vendor supplying a canned update image are nearly nil.
The more that these seething piles of crap can be hidden from the rest of the Internet, the better.
Posted May 10, 2017 16:17 UTC (Wed)
by raven667 (subscriber, #5198)
[Link]
I think you are right and its frustrating because unlike in the past, where the technology to secure devices wasn't built yet, people building IoT devices are taking tools which have access control, auth, etc. available and turning those features off, making by their action something reasonably secure, pathetic. In the same way that SSH raised the bar for default remote access/admin we need the same kind of effort to make the defaults for these kind of embedded devices reasonably decent. Basic netfilter rules and key-based auth using widely audited tools, could make these kinds of devices secure against the kind of wholesale attacks we are seeing.
Posted May 12, 2017 6:37 UTC (Fri)
by if.gnu.linux (guest, #88877)
[Link] (1 responses)
I think that there are two other sides we have to consider.
The first one is we as consumers are 'guilty' for always demanding new products which are more speedy, have more RAM etc. and for replacing one year old devices with new ones.
The second one is softwares need more RAM, more GPU power etc. to run. I think software developers think that "Nowadays a device has 4 GB RAM so what is wrong if my application use 1 GB of that RAM."
Posted May 12, 2017 11:16 UTC (Fri)
by pizza (subscriber, #46)
[Link]
Note that it is often *cheaper* to put more RAM in the device than less.
Posted May 4, 2017 6:38 UTC (Thu)
by Gollum (guest, #25237)
[Link] (4 responses)
The ME steals the packets off the NIC before the OS gets any opportunity to know about them. Which means that an OS based firewall will be oblivious to these connections. Otherwise, you could lock yourself out of your device by installing the firewall, and forgetting your creds. That would defeat the point of being able to manage even a powered off device (obviously, standby power is required).
Posted May 4, 2017 10:51 UTC (Thu)
by mikemol (guest, #83507)
[Link] (3 responses)
You're still at risk from your local "trusted" network, but then you always will be without things like isolated private VLANS or firewall rules applied at your switching layer and in your APs.
Posted May 4, 2017 11:19 UTC (Thu)
by Gollum (guest, #25237)
[Link]
Posted May 4, 2017 15:10 UTC (Thu)
by felixfix (subscriber, #242)
[Link] (1 responses)
Posted May 4, 2017 20:35 UTC (Thu)
by mikemol (guest, #83507)
[Link]
Hence a component of my remark:
> as many security layers as you reasonably can,
Keywords being *reasonably* and *layers*.
If you can't trust your network gateway, you can't trust your network gateway. If you find it a reasonable precaution, you might further segment your network such that the device you don't trust has an additional firewalled router between your clients and the untrusted device.
I do exactly this in one of my networks; I have to deal with routers on my premises owned and managed by AT&T and Comcast for upstream ISPs. Any traffic coming in on those first passes through a network segment for unfiltered traffic before they reach a router I own and control. And even on the *filtered* side of that router, there are routing nodes with stateful firewalls that segment, e.g. office activity traffic from server activity, server activity from various device management interfaces, etc. In the eventuality of a laptop or desktop infection, I don't need the infected code having easy access to nodes that the laptop or desktop shouldn't already have access to.
It's honestly not a difficult architecture to maintain, though it'd be overkill for home users.
Posted May 4, 2017 19:38 UTC (Thu)
by JanC_ (guest, #34940)
[Link] (7 responses)
Posted May 4, 2017 20:47 UTC (Thu)
by mikemol (guest, #83507)
[Link] (6 responses)
For every countermeasure, it can be assumed there's a counter-countermeasure; even DPI firewalls all the way down at the network's layer 2 could be worked around, depend on how the NIC views locally-destined packets in the outbound buffer.
But the countermeasure that defeats the vast majority of attacks is simply having a firewall between your node and the Internet that blocks unsolicited inbound connections.
Posted May 6, 2017 0:23 UTC (Sat)
by dlang (guest, #313)
[Link] (5 responses)
It's unreasonable for Intel or anyone else to ship something like this that can be accessed on any network interface, underneath the OS.
I fully understand the value of network management interfaces, but they should be on a separate interface that cannot mix with the traffic on the 'normal' interfaces.
Posted May 6, 2017 8:23 UTC (Sat)
by nix (subscriber, #2304)
[Link] (4 responses)
(It's probably really useful in the things' intended habitat, in the datacentre, where all the management LANs from a bunch of machines can end up on a physically distinct subnet from the actual traffic. It's not such a useful feature for the machine sitting by my desk :) I can understand why machines intended to be used in non-datacentre environments don't spend the extra on separate hardware for another physical network interface that will almost never be used.)
Intel do at least make timely security updates for the BMC and ME on machines they made the motherboard and chipset for available on the open Internet, making them the only vendors I have ever had experience with who will actually provide this sort of thing for people paying less than $BIGBUCKS for a support contract. (Or... as timely as you can expect for software that can brick large numbers of very expensive machines if it goes wrong enough.)
Posted May 10, 2017 7:24 UTC (Wed)
by joib (subscriber, #8541)
[Link] (3 responses)
E.g. Dell is one such biggish vendor that typically doesn't provide dedicated BMC ports.
So it's nice in that you don't need separate cables and switch ports for the management network, but I guess there's a security trade-off as well. Though I haven't heard of attacks that manage to cross to the "wrong" VLAN (which of course doesn't prove that such doesn't exist).
Posted May 10, 2017 15:03 UTC (Wed)
by nix (subscriber, #2304)
[Link] (2 responses)
Posted May 10, 2017 15:09 UTC (Wed)
by pizza (subscriber, #46)
[Link] (1 responses)
I have several SuperMicro BigTwin and FatTwin 4-node boxes that have dedicated ports for each node's BMC.
Posted May 12, 2017 12:44 UTC (Fri)
by nix (subscriber, #2304)
[Link]
Posted May 3, 2017 22:50 UTC (Wed)
by raven667 (subscriber, #5198)
[Link] (3 responses)
That's going in the wrong direction, we have a mobile devices, IoT, heading to a more connected future, these devices have to have the security tools builtin to protect themselves against the big bad internet without the crutch of VPNs and VLANs and NAT44. Anything running Linux should be modern enough to at least have a packet filter if not modern crypto, so we should demand simple robust security tools for these kinds of embedded long-lived devices, rather than just assuming that they are always on a "secure" network where there are never any malicious agents. If anything use IPv6 link-local addresses only for initial zero-touch provisioning, enroll key-pairs with your management systems, then you can drop the device anywhere on the internet with mutual authentication, safely.
Posted May 3, 2017 23:01 UTC (Wed)
by mikemol (guest, #83507)
[Link]
A host packet filter isn't going to help if your NIC is siphoning off packets for consumption by a management engine before it reaches the host network stack. You should certainly have such a filter, but you can't rely on it exclusively.
Posted May 9, 2017 5:48 UTC (Tue)
by ringerc (subscriber, #3071)
[Link] (1 responses)
That'd be nice. But the opposite is happening. There are no economic incentives to make secure consumer routers, IoT devices, and appliances. There are huge incentives to save development costs, and entirely forego software support for the lifetime of the product. In the world of the 30 day warranty (oh, how glad I am for Australia's legally mandated 1 year minimum warrenty), security holes in your fridge aren't even on the radar.
These things need to be treated like the plague-bearing insecure and un-secureable monstrosities they are, until or unless the economic incentives applied by buyers to vendors change that.
This becomes a problem when their functionality requires them to:
* Accept input, or even new code to run, from "trusted" Internet hosts, which often mean "whatever updates.mycompany.com resolves to and replies on port 80"
or, often as not, when their functionality doesn't require that, but they do it anyway.
I honestly won't be surprised if we start seeing household firewall products (both discrete and home-router-embedded) that proxy your IoT crap behind a secondary wifi network, filtering out the exploits and limiting their access.
The real worry is the ones that talk directly to public WiFi, to 3G/4G cellular networks, to hotel networks, etc, where there is no trust. You could make an Ethernet/wifi condom (proxy/filter) device for ones that use WiFi or Ethernet, but not really for cellular.
I have no idea how to help change the situation to create meaningful incentives for secure device development and deployment. But without it, the whole Internet's going to be in trouble.
Posted May 11, 2017 10:52 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
For those who haven't heard, a ?white?hat wrote a worm. It targeted several common IoT devices, and did NOT try very hard to break in. When it did, it tried to switch on as many security features as it could before self-destructing.
If it *couldn't* secure the device, it overwrote the firmware with /dev/zero and rebooted...
Apparently it's made a massive dent in the current round of DDos attacks :-)
Cheers,
Posted May 3, 2017 20:59 UTC (Wed)
by nix (subscriber, #2304)
[Link]
Posted May 4, 2017 3:03 UTC (Thu)
by pabs (subscriber, #43278)
[Link] (2 responses)
Posted May 4, 2017 4:38 UTC (Thu)
by mjg59 (subscriber, #23239)
[Link] (1 responses)
Posted May 4, 2017 8:27 UTC (Thu)
by corsac (subscriber, #49696)
[Link]
Posted May 4, 2017 5:56 UTC (Thu)
by epa (subscriber, #39769)
[Link]
Posted May 4, 2017 9:29 UTC (Thu)
by arekm (guest, #4846)
[Link] (3 responses)
> the "second processor" to which Greve referred. It is present on all Intel's current chipsets
Is the second cpu in (intel) cpu or in (motherboard intel) chipset?
Posted May 4, 2017 9:37 UTC (Thu)
by zdzichu (subscriber, #17118)
[Link] (2 responses)
For the record, AMD equivalent of this (Platform Security Processor) is actually an ARM core embedded in x86_64 CPU itself.
Posted May 4, 2017 16:21 UTC (Thu)
by rahvin (guest, #16953)
[Link] (1 responses)
Posted May 6, 2017 8:14 UTC (Sat)
by marcH (subscriber, #57642)
[Link]
It's not very clear what you meant here by "firmware".
Actually, if you hadn't given context then "CPU" wouldn't have been very clear either. I think you meant "SoC".
From https://en.wikipedia.org/wiki/Central_processing_unit
Posted May 5, 2017 20:01 UTC (Fri)
by corsac (subscriber, #49696)
[Link]
TL;DR: HTTPDigest authentication uses strncmp() with size provided by remote party.
Posted May 7, 2017 13:10 UTC (Sun)
by mb (subscriber, #50428)
[Link] (2 responses)
$ lspci|egrep -i 'mei|heci'
Posted May 7, 2017 13:47 UTC (Sun)
by karkhaz (subscriber, #99844)
[Link]
> If it does show an MEI controller, that still doesn't mean you're vulnerable - AMT may still not be provisioned.
I started the configuration menu (pressing ctrl+p at boot, then supplying the default password of `admin,' then changing the password to something that the utility found acceptable. This implies that nobody has used this menu before on my laptop (I bought it second-hand)). There was an option about "unprovisioning" AMT, which I selected, and nothing much seemed to change; after it finished, there was still the same menu item allowing me to unprovision. So I then just rebooted the machine. I couldn't see anything to indicate whether AMT actually _is_ provisioned in that BIOS menu. A bit confusing.
Posted May 7, 2017 13:53 UTC (Sun)
by pabs (subscriber, #43278)
[Link]
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
> Pathetically insecure devices that never get security audits or updates
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
AFAIK it's actually fairly common to not have a separate RJ45 port for the BMC, but rather the BMC is connected to one or more of the onboard RJ45 ports and shared with the host OS. The BMC does have it's own MAC though, and standard practice is to use a different VLAN for the management network.
Oh indeed: I've only ever owned one machine with a dedicated BMC RJ45 out of several with BMCs. I'm not aware of any machines on which the BMC is *only* connected to a dedicated RJ45, though -- they all seem to be able to share NICs with the main CPU (and indeed are usually configured that way by default).
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
> I know there are people who've put their management ports on public IPs.
> But still. *Jeeze*. Don't Do That!
> Your management interfaces belong on private ports (or VLANs, if you must) on private broadcast domains reachable *only* via authenticated, secured VPN.
Intel's zero-day problem
Intel's zero-day problem
* Use UPnP to accept inbound network requests from arbitrary Internet hosts
* Uncritically communicate with anything on the local network segment or anything directly routable and addressible from it
Intel's zero-day problem
Wol
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Intel's zero-day problem
Specifically: (this is all from Wiki using the parent link)Intel's zero-day problem
>Until the release of Nehalem processors, the ME was usually embedded into the motherboard's northbridge, following the Memory Controller Hub (MCH) layout.[37] With the newer Intel architectures (Intel 5 Series onwards), ME is included into the Platform Controller Hub (PCH).[38][39]
The vulnerable processors are Nehalem (2008) and above, this means all vulnerable systems are the ones using the PCH, the replacement for the old north/south system after the northbridge was incorporated into the CPU. This is the largest chip on the motherboard that's not the CPU. Thank you for posting this, I was under the impression the ME was incorporated into the firmware on the CPU.
For those that are curious what the mangement engine can do:
Hardware-based AMT features on laptop and desktop PCs include:
Laptops with AMT also include wireless technologies:
Oh and the ME is active even if the PC isn't powered on.
Intel's zero-day problem
> An IC that contains a CPU may also contain memory, peripheral interfaces, and other components of a computer [like... some Management Engine?]; such integrated devices are variously called microcontrollers or systems on a chip (SoC).
Intel's zero-day problem
Intel's zero-day problem
00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 (rev 04)
Intel's zero-day problem
Intel's zero-day problem