PCIe, power management, and problematic BIOSes [LWN.net]

By Jonathan Corbet
June 29, 2011

Back in April, Phoronix announced with some fanfare that the 2.6.38 kernel - and those following - had a "major" power management regression which significantly reduced battery life on some systems. This problem has generated a fair amount of discussion, including this Launchpad thread, but little in the way of solutions. Phoronix now claims to have located the change that caused the problem and has provided a workaround which will make things better for some users. But a true fix may be a while in coming.

As a result of the high clock rates used, PCI-Express devices can take a lot of power even when they are idle. "Active state power management" (ASPM) was developed as a means for putting those peripherals into a lower power state when it seems that there may be little need for them. ASPM can save power, but the usual tradeoff applies: a device which is in a reduced power state will not be immediately available for use. So, on systems where ASPM is in use, access to devices can sometimes take noticeably longer if those devices have been powered down. In some situations (usually those involving batteries) this tradeoff may be acceptable; in others it is not. So, like most power management mechanisms, ASPM can be turned on or off.

It is a bit more complicated than that, though; on x86 systems, the BIOS also gets into the act. The BIOS is charged with the initial configuration of the system and with telling the kernel about the functionality that is present. One bit of information passed to the kernel by the BIOS is whether ASPM is supported on the system. The kernel developers, reasonably, concluded that, if the BIOS says that ASPM is not supported, they should not mess with the associated registers. It turns out, though, that this approach didn't quite work; thus, in December, Matthew Garrett committed a patch described as:

We currently refuse to touch the ASPM registers if the BIOS tells us that ASPM isn't supported. This can cause problems if the BIOS has (for any reason) enabled ASPM on some devices anyway. Change the code such that we explicitly clear ASPM if the FADT indicates that ASPM isn't supported, and make sure we tidy up appropriately on device removal in order to deal with the hotplug case.

In other words, sometimes the BIOS will tell the system that ASPM is not supported even though ASPM support is present; for added fun, the BIOS may enable ASPM on some devices (even though it says ASPM is not supported) before passing control to the kernel. There are reasons why operating system developers tend to hold BIOS developers in low esteem.

Had Andrew Morton read the above changelog, he certainly would have complained that "can cause problems" is a rather vague justification for a change to the kernel. Your editor asked Matthew about the problem and got an informative response that is worth reading in its entirety:

If this bit is set, the platform is indicating to the OS that it doesn't support ASPM. In the past we took that to mean that we simply shouldn't touch the ASPM bits. However, it turns out that there's some systems where the BIOS has enabled ASPM itself, set the "ASPM unsupported" bit and then the hardware falls over when an ASPM transition occurs. The most straightforward thing to assume was that the BIOS was stupid (which is, to be fair, my default assumption) and shouldn't have enabled ASPM. So, since that patch, we clear the ASPM state when the BIOS indicates that the platform doesn't support ASPM.

It's not hard to imagine that putting devices into a state that the kernel was told should not exist might create confusion somewhere. Some research turns up, for example, this bug report about system hangs which are fixed by Matthew's patch. If the BIOS says that ASPM is not supported, it would seem that ensuring that no devices think otherwise would make sense.

That said, this patch is the one that the bisection effort at Phoronix has fingered as the cause of the power regression. Apparently, the notion that disabling low-power states in hardware may lead to increased power consumption also makes sense. The workaround suggested in the article is to boot with the pcie_aspm=force option; that forces the system to turn on ASPM regardless of whether the BIOS claims to support it. This workaround will undoubtedly yield better battery life on some affected systems; others may well not work at all. In the latter case, the system may simply lock up - a state with even worse latency characteristics combined with surprisingly bad power use. So this workaround may be welcomed by users who have seen their battery life decline significantly, but it is not a proper solution to the problem.

Finding that proper solution - preferably one which Just Works without any need for special boot parameters - could be tricky. Quoting Matthew again:

What alternatives are there? We could keep the status quo and add driver whitelisting for hardware setups that are known to work. The problem is that even where we have specifications for the hardware, we often don't have the errata lists. We don't know for sure whether it works or not. We could revert this patch and add more driver blacklisting. But then we need to track down every device that doesn't work. Or, it's possible that the original code was correct and Linux simply programs the hardware differently, triggering ASPM issues that aren't seen elsewhere.

Given the uncertainty in the situation, the kernel developers have reached the conclusion that "waste a bit of power" is a lesser evil than "lock up on some systems." In the absence of a better understanding of the problem, any other approach would be hard to justify. So some users may have to use the pcie_aspm=force workaround for a while yet.

Meanwhile, the power usage problem has, as far as your editor can tell, never been raised on any kernel development mailing list. It never appeared in the 2.6.38 regression list. So this issue was invisible to much of the development community; it's not entirely surprising that it has not received much in the way of attention from developers. For better or for worse, the development community has its way of dealing with issues. Reporting a bug to linux-kernel certainly does not guarantee that it will get fixed, but it does improve the odds considerably. Had this issue been brought directly to the developers involved, we might have learned about the root cause some time ago.

(Log in to post comments)

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 18:13 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

BIOS authors typically end up with documentation from hardware vendors and a Leading Alternative OS vendor which describe their expectations of the BIOS, and so BIOSes are often written based on those assumptions. The problem is that Linux vendors (even large ones) rarely get to see any of this documentation, even if they have NDAs with the hardware vendors. We're usually left trying to determine the nature of the expected behaviour by looking at shadows cast on walls. So the answer to your question may be "Because the hardware and OS vendor told them to", or it may be "Because their crack was cut with meth that morning". We just don't know.

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 18:16 UTC (Wed) by flewellyn (subscriber, #5047) [Link]

I see. So, is there a way for the OS to test for this ASPM support without recourse to the BIOS?

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 18:28 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

Not really. In an ideal world hardware that broke with ASPM simply wouldn't advertise ASPM support, but doing that was actually a spec violation until recently. So you're left with hardware advertising ASPM support and you have no way of knowing whether or not it's safe to touch that, and in some cases *turning off* ASPM can cause hardware to explode as well. So you either need complete knowledge of all hardware where ASPM is broken, or you need to assume that the BIOS will give you reliable information as to whether or not ASPM works on that platform.

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 20:06 UTC (Wed) by samroberts (subscriber, #46749) [Link]

But how does the BIOS know whether ASPM works or not?

If it figures out from the devices, the kernel could do this, too.

Since you say the kernel can't do this reliably, I assume the BIOS authors have to hard-code this information about the devices? I guess, in theory, they should know, but it sounds pretty horrible.

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 20:13 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

Because the BIOS has been tested with that set of hardware. This is typically a laptop problem, not a desktop one.

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 20:24 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]

because the bios engineer hard-coded it to say if it works or not.

the BIOS isn't testing things and deciding it doesn't work, it's a hard-coded entry made by the BIOS programmer.

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 20:40 UTC (Wed) by flewellyn (subscriber, #5047) [Link]

So we have a situation in which the firmware lies to you, and then penalizes you if you believe it? What a wonderful world.

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 21:00 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

Not an uncommon situation, sadly.

PCIe, power management, and problematic BIOSes

Posted Jun 29, 2011 22:17 UTC (Wed) by bronson (subscriber, #4806) [Link]

You're too kind Matthew. To engineer a BIOS, most companies copy something from last year, stab it until it boots, and ship it.

Check out the top vendor on http://smolts.org . Or http://smolts.org/reports/view_profiles?profile=Filled&...

Not much craftsmanship to be found.

PCIe, power management, and problematic BIOSes

Posted Jun 30, 2011 0:53 UTC (Thu) by AndreE (subscriber, #60148) [Link]

So does the Leading Alternative OS have greater access to documentation and vendor information?

I'm can't see how they themselves could solve this problem without having a perfect blacklist of broken motherboards.

PCIe, power management, and problematic BIOSes

Posted Jun 30, 2011 1:46 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]

Microsoft generally doesn't have this specific information; it relies on OEMs to do the integration.

The OEMs buy in hardware components, which come with Windows drivers already written, and they buy firmware from other companies. Normally they get some specific hardware documentation and source code for the drivers and firmware. They also have documentation and tools for Windows (the DDK).

They need to make the combination of hardware, firmware, OS and drivers work, and quickly. Since they control both firmware and driver code (except for generic drivers provided by Microsoft) they may resort to dirty hacks in either or both of these. The result may work only as a result of undocumented behaviour of Windows.

The challenge for free OS developers is to replicate these driver hacks and undocumented behaviour of Windows where necessary.

PCIe, power management, and problematic BIOSes

Posted Jun 30, 2011 7:26 UTC (Thu) by ebirdie (subscriber, #512) [Link]

>Since they control both firmware and driver code (except for generic drivers provided by Microsoft) they may resort to dirty hacks in either or both of these. The result may work only as a result of undocumented behaviour of Windows.

I see! That explains the phenomenon we have wondered for years, when working as Windows reinstaller junkies. The question asked, what makes a decent hardware become so unusable with Windows over short and longer periods of time?

Now I understand that the frequency on patching Windows also ruins the underlaying assuptions made when producing "a nice working decent hardware". The common solution is to fix the hardware (slow hardware, sleep states working poorly, poor battery duration, a device not working after awake, irritating moments in prepared presentations where reboot is often the cure to rescue the presentation etc. etc. and after the presentation the laptop is returned for "repair") with buying new hardware, what keeps the wheels going for the group in the party.

PCIe, power management, and problematic BIOSes

Posted Jun 30, 2011 19:29 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]

Note, there are some limiting factors on OEM hacks. If the OEM installs the 64-bit edition of Windows or if they use the 'Designed for Windows' logo then all drivers must pass a fairly challenging test suite (WHQL test). So there is some disincentive to modifying drivers. I'm not sure whether there is a similar test suite for the firmware.

PCIe, power management, and problematic BIOSes

Posted Jul 6, 2011 0:59 UTC (Wed) by cortana (subscriber, #24596) [Link]

Is it true that some drivers will detect that they are being put through the WHQL process, and modify their behaviour so that they pass, but act in a way that they would never do in the real world?

PCIe, power management, and problematic BIOSes

Posted Jul 6, 2011 1:21 UTC (Wed) by BenHutchings (subscriber, #37955) [Link]

Since the WHQL test is run by the submitter, not Microsoft, there are any number of ways to cheat. I assume there are penalties for companies that get caught doing this.

PCIe, power management, and problematic BIOSes

Posted Jul 6, 2011 12:09 UTC (Wed) by nye (guest, #51576) [Link]

>Is it true that some drivers will detect that they are being put through the WHQL process, and modify their behaviour so that they pass, but act in a way that they would never do in the real world?

If they know how they're supposed to act, why would they deliberately act differently in normal circumstances?

PCIe, power management, and problematic BIOSes

Posted Jul 7, 2011 17:13 UTC (Thu) by farnz (guest, #17727) [Link]

As an example; DirectX has a DoesDriverSupport method that it calls to see what functionality the driver supports. It's obvious that an implementation that always returns TRUE is faster than one that returns an accurate result.

Less obvious, but still true, is that a driver that can currently support everything the platform uses can return TRUE without checking, and will be a tiny bit faster. There are similar cases throughout any significant sized API, where being wrong happens to work for today's software, and is faster because you do less work - and when those cases are on the fastpath, the driver will do them.

WHQL tries to deliberately break these sorts of things - it looks for cases where the answer can be predicted, and checks that the driver gives the right answer; if it's lying, WHQL will break things.

Hypothetically, for example, imagine that your GPU only has a single thread of execution, used for 3D commands and for putting buffers in the hands of scanout to display, but lets you access buffers from the CPU directly, bypassing the GPU execution. A driver could implement glXSwapBuffers and friends by putting the swap in the GPU's thread of execution, and returning immediately; it could then make glFinish and glFlush no-ops, and not break anything obvious. If Microsoft thought drivers were doing this sort of trick, WHQL could do a glReadPixels immediately after a glFinish, and get the wrong result - the driver's been caught lying.

In the meantime, of course, the driver is faster than the competition's driver in benchmarks people care about - because it's not doing things by the spec, and hoping that you'll never notice the lie.

PCIe, power management, and problematic BIOSes

Posted Jul 12, 2011 13:06 UTC (Tue) by nye (guest, #51576) [Link]

>As an example; DirectX has a DoesDriverSupport method that it calls to see what functionality the driver supports. It's obvious that an implementation that always returns TRUE is faster than one that returns an accurate result.

That would presumably be caught by some things visibly breaking at some point, otherwise there's no point in having it in the first place. (I wonder what the modified version does in that example when it catches the driver lying.)

>Less obvious, but still true, is that a driver that can currently support everything the platform uses can return TRUE without checking, and will be a tiny bit faster. There are similar cases throughout any significant sized API, where being wrong happens to work for today's software, and is faster because you do less work - and when those cases are on the fastpath, the driver will do them.

This does at least make more sense - if it definitely isn't causing any problems now, then I can imagine somebody saying 'we can always update it in the future' - and possibly even believing it.

>In the meantime, of course, the driver is faster than the competition's driver in benchmarks people care about - because it's not doing things by the spec, and hoping that you'll never notice the lie

One might hope that driver authors would expect people to care whether their very fast driver is unstable or has rendering glitches, and if they have a more accurate WHQL-passing driver (as posited upthread) to provide that as an option.

*Sigh*

I guess worse things happen at sea.

PCIe, power management, and problematic BIOSes

Posted Jul 12, 2011 13:56 UTC (Tue) by farnz (guest, #17727) [Link]

In Raymond's example, the implementation handles a detected lie by assuming that DoesDriverSupport always returns FALSE, and not using the accelerated paths. In other words, if you're ever caught lying, you're never going to be trusted to do anything sophisticated, even if you could do some acceleration.

Unfortunately, too many people buy hardware on the basis of benchmarks - for an example, look at the QUACK.EXE incident - a GPU driver was set up to detect a specific application used as a benchmark, and cheat.

The problem for buyers of devices with complex drivers is that until you work out what the cheats are, you don't know whether the driver is fast in benchmarks because it cheats, or because it's buggy, or because it's genuinely that fast, or because your applications are buggy and relying on things not guaranteed by the API. Add in closed-source drivers, which can do things like detect the presence of WHQL certification tests on the machine, and you end up with a driver that (for example) is slow and stable when you run the WHQL test suite (thus always passes), but takes shortcuts when WHQL is not running. As benchmarkers rarely have WHQL installed, the driver author gets the "best" of both worlds - stability if you try and test it with WHQL (so you have a WHQL-compliant driver), and fast if you try and benchmark it without WHQL.

Now throw in the idea that applications don't use complex functionality at first, and you see just how painful things can get - the bit that fails on you might be something that no application today uses, at which point it can be years before anyone writes test code that shows the problem is the driver. For some classes of driver (e.g. graphics drivers), people build up a whole set of mythology around things you cannot do, and you develop a set of shared assumptions that aren't actually in the spec, but that "everyone knows" are things that don't work, because drivers traditionally cheated.

same for everybody, not just us

Posted Jul 1, 2011 9:47 UTC (Fri) by tialaramex (subscriber, #21167) [Link]

Also keep in mind that we're approaching, if not already at, the point where "weird bugs with Windows" aren't that much less common than "weird bugs with Linux". Notice I say "with" not "in" because often it's just as likely that the hardware is faulty and the "working" OS just happens not to trigger the fault as is probably the case with this ASPM problem.

My current laptop "forgets" it has built-in speakers if suspended with the headphones plugged in while running Windows. Only a reboot _into Fedora_ fixes the problem (rebooting just Windows has no effect).

The outwardly normal USB to SATA/IDE drive cases I bought recently turn out to occasionally give erroneous read results with IDE drives on Windows. The data on disk is undamaged, but understandably that's not good enough for me. Can't reproduce in Linux. If this had been the other way around, would the retailer have taken them back as faulty? I'm glad I don't have to find out.

PCIe, power management, and problematic BIOSes

Posted Jun 30, 2011 17:08 UTC (Thu) by iceblink (subscriber, #19982) [Link]

In at least one BIOS that I have worked with the ACPI spec was simply mis-interpreted as needing this flag to be set it to 1 in order to enable ASPM. Seems like a silly mistake but I suspect it is common.

The kernel should probably also be checking to see if the FACS claims to support ACPI 4.0 before relying on this flag.

PCIe, power management, and problematic BIOSes

Posted Jun 30, 2011 17:24 UTC (Thu) by mjg59 (subscriber, #23239) [Link]

Did anyone test that ASPM actually ended up enabled as a result? You're right that the check should be made conditional on the FADT version, though.

PCIe, power management, and problematic BIOSes

Posted Jul 2, 2011 2:05 UTC (Sat) by iceblink (subscriber, #19982) [Link]

Yeah we noticed a spike in power testing and ended up with a DMI workaround until BIOS updates can be rolled out: http://crosbug.com/15124

PCIe, power management, and problematic BIOSes

Posted Jul 2, 2011 3:05 UTC (Sat) by mjg59 (subscriber, #23239) [Link]

Ah, ok, I hadn't realised this was Chrome OS. My concern is more that BIOS vendors in general have this opinion, and Windows handles it differently to us in some way.