PCIe, power management, and problematic BIOSes
As a result of the high clock rates used, PCI-Express devices can take a lot of power even when they are idle. "Active state power management" (ASPM) was developed as a means for putting those peripherals into a lower power state when it seems that there may be little need for them. ASPM can save power, but the usual tradeoff applies: a device which is in a reduced power state will not be immediately available for use. So, on systems where ASPM is in use, access to devices can sometimes take noticeably longer if those devices have been powered down. In some situations (usually those involving batteries) this tradeoff may be acceptable; in others it is not. So, like most power management mechanisms, ASPM can be turned on or off.
It is a bit more complicated than that, though; on x86 systems, the BIOS also gets into the act. The BIOS is charged with the initial configuration of the system and with telling the kernel about the functionality that is present. One bit of information passed to the kernel by the BIOS is whether ASPM is supported on the system. The kernel developers, reasonably, concluded that, if the BIOS says that ASPM is not supported, they should not mess with the associated registers. It turns out, though, that this approach didn't quite work; thus, in December, Matthew Garrett committed a patch described as:
In other words, sometimes the BIOS will tell the system that ASPM is not supported even though ASPM support is present; for added fun, the BIOS may enable ASPM on some devices (even though it says ASPM is not supported) before passing control to the kernel. There are reasons why operating system developers tend to hold BIOS developers in low esteem.
Had Andrew Morton read the above changelog, he certainly would have complained that "can cause problems" is a rather vague justification for a change to the kernel. Your editor asked Matthew about the problem and got an informative response that is worth reading in its entirety:
It's not hard to imagine that putting devices into a state that the kernel was told should not exist might create confusion somewhere. Some research turns up, for example, this bug report about system hangs which are fixed by Matthew's patch. If the BIOS says that ASPM is not supported, it would seem that ensuring that no devices think otherwise would make sense.
That said, this patch is the one that the bisection effort at Phoronix has fingered as the cause of the power regression. Apparently, the notion that disabling low-power states in hardware may lead to increased power consumption also makes sense. The workaround suggested in the article is to boot with the pcie_aspm=force option; that forces the system to turn on ASPM regardless of whether the BIOS claims to support it. This workaround will undoubtedly yield better battery life on some affected systems; others may well not work at all. In the latter case, the system may simply lock up - a state with even worse latency characteristics combined with surprisingly bad power use. So this workaround may be welcomed by users who have seen their battery life decline significantly, but it is not a proper solution to the problem.
Finding that proper solution - preferably one which Just Works without any need for special boot parameters - could be tricky. Quoting Matthew again:
Given the uncertainty in the situation, the kernel developers have reached the conclusion that "waste a bit of power" is a lesser evil than "lock up on some systems." In the absence of a better understanding of the problem, any other approach would be hard to justify. So some users may have to use the pcie_aspm=force workaround for a while yet.
Meanwhile, the power usage problem has, as far as your editor can tell,
never been raised on any kernel development mailing list. It never
appeared in the 2.6.38 regression list. So this issue was invisible to much
of the development community; it's not entirely surprising that it has not
received much in the way of attention from developers. For better or for
worse, the development community has its way of dealing with issues.
Reporting a bug to linux-kernel certainly does not guarantee that it will
get fixed, but it does improve the odds considerably. Had this issue been
brought directly to the developers involved, we might have learned about
the root cause some time ago.
Index entries for this article | |
---|---|
Kernel | Power management |
Posted Jun 29, 2011 16:45 UTC (Wed)
by cesarb (subscriber, #6266)
[Link] (3 responses)
printk(KERN_INFO"ACPI FADT declares the system doesn't support PCIe ASPM, so disable it\n");
So, if that message is in your dmesg, you might be affected.
Posted Jun 29, 2011 18:12 UTC (Wed)
by troy.unrau (guest, #73654)
[Link] (2 responses)
# dmesg | grep ASPM
I'm not sure this means that the kernel has decided to turn it off anyway, or if my laptop truly doesn't support it. Still need more data to determine if power drain improves.
Posted Jun 29, 2011 18:18 UTC (Wed)
by cesarb (subscriber, #6266)
[Link]
+void pcie_clear_aspm(void)
Posted Jun 29, 2011 18:30 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link]
Posted Jun 29, 2011 16:53 UTC (Wed)
by nix (subscriber, #2304)
[Link] (2 responses)
So far the e1000e guys have declared defeat and I've been too lazy/busy to hand it off to the PCI guys for further diagnosis (in part because the sourceforge bugtracker is so bloody ugly that I'm mildly ashamed to point anyone at it. Oops.)
Posted Jun 29, 2011 16:57 UTC (Wed)
by cesarb (subscriber, #6266)
[Link] (1 responses)
Posted Jun 30, 2011 12:35 UTC (Thu)
by nix (subscriber, #2304)
[Link]
Posted Jun 29, 2011 18:05 UTC (Wed)
by flewellyn (subscriber, #5047)
[Link] (25 responses)
This raises the question of exactly why a BIOS would do such an asinine thing. Just what is it about BIOS development that creates such...um...creatively broken code?
Posted Jun 29, 2011 18:13 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link] (20 responses)
Posted Jun 29, 2011 18:16 UTC (Wed)
by flewellyn (subscriber, #5047)
[Link] (6 responses)
Posted Jun 29, 2011 18:28 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link] (5 responses)
Posted Jun 29, 2011 20:06 UTC (Wed)
by samroberts (subscriber, #46749)
[Link] (2 responses)
If it figures out from the devices, the kernel could do this, too.
Since you say the kernel can't do this reliably, I assume the BIOS authors have to hard-code this information about the devices? I guess, in theory, they should know, but it sounds pretty horrible.
Posted Jun 29, 2011 20:13 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link]
Posted Jun 29, 2011 20:24 UTC (Wed)
by dlang (guest, #313)
[Link]
the BIOS isn't testing things and deciding it doesn't work, it's a hard-coded entry made by the BIOS programmer.
Posted Jun 29, 2011 20:40 UTC (Wed)
by flewellyn (subscriber, #5047)
[Link] (1 responses)
Posted Jun 29, 2011 21:00 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link]
Posted Jun 29, 2011 22:17 UTC (Wed)
by bronson (subscriber, #4806)
[Link]
Check out the top vendor on http://smolts.org . Or http://smolts.org/reports/view_profiles?profile=Filled&...
Not much craftsmanship to be found.
Posted Jun 30, 2011 0:53 UTC (Thu)
by AndreE (guest, #60148)
[Link] (10 responses)
I'm can't see how they themselves could solve this problem without having a perfect blacklist of broken motherboards.
Posted Jun 30, 2011 1:46 UTC (Thu)
by BenHutchings (subscriber, #37955)
[Link] (8 responses)
Microsoft generally doesn't have this specific information; it relies on OEMs to do the integration. The OEMs buy in hardware components, which come with Windows drivers already written, and they buy firmware from other companies. Normally they get some specific hardware documentation and source code for the drivers and firmware. They also have documentation and tools for Windows (the DDK). They need to make the combination of hardware, firmware, OS and drivers work, and quickly. Since they control both firmware and driver code (except for generic drivers provided by Microsoft) they may resort to dirty hacks in either or both of these. The result may work only as a result of undocumented behaviour of Windows. The challenge for free OS developers is to replicate these driver hacks and undocumented behaviour of Windows where necessary.
Posted Jun 30, 2011 7:26 UTC (Thu)
by ebirdie (guest, #512)
[Link] (7 responses)
I see! That explains the phenomenon we have wondered for years, when working as Windows reinstaller junkies. The question asked, what makes a decent hardware become so unusable with Windows over short and longer periods of time?
Now I understand that the frequency on patching Windows also ruins the underlaying assuptions made when producing "a nice working decent hardware". The common solution is to fix the hardware (slow hardware, sleep states working poorly, poor battery duration, a device not working after awake, irritating moments in prepared presentations where reboot is often the cure to rescue the presentation etc. etc. and after the presentation the laptop is returned for "repair") with buying new hardware, what keeps the wheels going for the group in the party.
Posted Jun 30, 2011 19:29 UTC (Thu)
by BenHutchings (subscriber, #37955)
[Link] (6 responses)
Posted Jul 6, 2011 0:59 UTC (Wed)
by cortana (subscriber, #24596)
[Link] (5 responses)
Posted Jul 6, 2011 1:21 UTC (Wed)
by BenHutchings (subscriber, #37955)
[Link]
Posted Jul 6, 2011 12:09 UTC (Wed)
by nye (subscriber, #51576)
[Link] (3 responses)
If they know how they're supposed to act, why would they deliberately act differently in normal circumstances?
Posted Jul 7, 2011 17:13 UTC (Thu)
by farnz (subscriber, #17727)
[Link] (2 responses)
As an example; DirectX has a DoesDriverSupport method that it calls to see what functionality the driver supports. It's obvious that an implementation that always returns TRUE is faster than one that returns an accurate result. Less obvious, but still true, is that a driver that can currently support everything the platform uses can return TRUE without checking, and will be a tiny bit faster. There are similar cases throughout any significant sized API, where being wrong happens to work for today's software, and is faster because you do less work - and when those cases are on the fastpath, the driver will do them.
WHQL tries to deliberately break these sorts of things - it looks for cases where the answer can be predicted, and checks that the driver gives the right answer; if it's lying, WHQL will break things.
Hypothetically, for example, imagine that your GPU only has a single thread of execution, used for 3D commands and for putting buffers in the hands of scanout to display, but lets you access buffers from the CPU directly, bypassing the GPU execution. A driver could implement glXSwapBuffers and friends by putting the swap in the GPU's thread of execution, and returning immediately; it could then make glFinish and glFlush no-ops, and not break anything obvious. If Microsoft thought drivers were doing this sort of trick, WHQL could do a glReadPixels immediately after a glFinish, and get the wrong result - the driver's been caught lying.
In the meantime, of course, the driver is faster than the competition's driver in benchmarks people care about - because it's not doing things by the spec, and hoping that you'll never notice the lie.
Posted Jul 12, 2011 13:06 UTC (Tue)
by nye (subscriber, #51576)
[Link] (1 responses)
That would presumably be caught by some things visibly breaking at some point, otherwise there's no point in having it in the first place. (I wonder what the modified version does in that example when it catches the driver lying.)
>Less obvious, but still true, is that a driver that can currently support everything the platform uses can return TRUE without checking, and will be a tiny bit faster. There are similar cases throughout any significant sized API, where being wrong happens to work for today's software, and is faster because you do less work - and when those cases are on the fastpath, the driver will do them.
This does at least make more sense - if it definitely isn't causing any problems now, then I can imagine somebody saying 'we can always update it in the future' - and possibly even believing it.
>In the meantime, of course, the driver is faster than the competition's driver in benchmarks people care about - because it's not doing things by the spec, and hoping that you'll never notice the lie
One might hope that driver authors would expect people to care whether their very fast driver is unstable or has rendering glitches, and if they have a more accurate WHQL-passing driver (as posited upthread) to provide that as an option.
*Sigh*
I guess worse things happen at sea.
Posted Jul 12, 2011 13:56 UTC (Tue)
by farnz (subscriber, #17727)
[Link]
In Raymond's example, the implementation handles a detected lie by assuming that DoesDriverSupport always returns FALSE, and not using the accelerated paths. In other words, if you're ever caught lying, you're never going to be trusted to do anything sophisticated, even if you could do some acceleration.
Unfortunately, too many people buy hardware on the basis of benchmarks - for an example, look at the QUACK.EXE incident - a GPU driver was set up to detect a specific application used as a benchmark, and cheat.
The problem for buyers of devices with complex drivers is that until you work out what the cheats are, you don't know whether the driver is fast in benchmarks because it cheats, or because it's buggy, or because it's genuinely that fast, or because your applications are buggy and relying on things not guaranteed by the API. Add in closed-source drivers, which can do things like detect the presence of WHQL certification tests on the machine, and you end up with a driver that (for example) is slow and stable when you run the WHQL test suite (thus always passes), but takes shortcuts when WHQL is not running. As benchmarkers rarely have WHQL installed, the driver author gets the "best" of both worlds - stability if you try and test it with WHQL (so you have a WHQL-compliant driver), and fast if you try and benchmark it without WHQL.
Now throw in the idea that applications don't use complex functionality at first, and you see just how painful things can get - the bit that fails on you might be something that no application today uses, at which point it can be years before anyone writes test code that shows the problem is the driver. For some classes of driver (e.g. graphics drivers), people build up a whole set of mythology around things you cannot do, and you develop a set of shared assumptions that aren't actually in the spec, but that "everyone knows" are things that don't work, because drivers traditionally cheated.
Posted Jul 1, 2011 9:47 UTC (Fri)
by tialaramex (subscriber, #21167)
[Link]
My current laptop "forgets" it has built-in speakers if suspended with the headphones plugged in while running Windows. Only a reboot _into Fedora_ fixes the problem (rebooting just Windows has no effect).
The outwardly normal USB to SATA/IDE drive cases I bought recently turn out to occasionally give erroneous read results with IDE drives on Windows. The data on disk is undamaged, but understandably that's not good enough for me. Can't reproduce in Linux. If this had been the other way around, would the retailer have taken them back as faulty? I'm glad I don't have to find out.
Posted Sep 28, 2016 13:15 UTC (Wed)
by Hi-Angel (guest, #110915)
[Link]
Posted Jun 30, 2011 17:08 UTC (Thu)
by iceblink (guest, #19982)
[Link] (3 responses)
The kernel should probably also be checking to see if the FACS claims to support ACPI 4.0 before relying on this flag.
Posted Jun 30, 2011 17:24 UTC (Thu)
by mjg59 (subscriber, #23239)
[Link] (2 responses)
Posted Jul 2, 2011 2:05 UTC (Sat)
by iceblink (guest, #19982)
[Link] (1 responses)
Posted Jul 2, 2011 3:05 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link]
Posted Jun 30, 2011 10:08 UTC (Thu)
by liljencrantz (guest, #28458)
[Link] (2 responses)
Posted Jun 30, 2011 14:55 UTC (Thu)
by mjg59 (subscriber, #23239)
[Link] (1 responses)
Posted Jul 3, 2011 9:25 UTC (Sun)
by liljencrantz (guest, #28458)
[Link]
Words fail me.
Posted Jun 30, 2011 18:16 UTC (Thu)
by robert_s (subscriber, #42402)
[Link] (6 responses)
Posted Jul 2, 2011 14:25 UTC (Sat)
by Lennie (subscriber, #49641)
[Link] (5 responses)
Posted Jul 2, 2011 14:35 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link] (4 responses)
Posted Jul 2, 2011 18:30 UTC (Sat)
by Lennie (subscriber, #49641)
[Link] (3 responses)
Posted Jul 2, 2011 18:54 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link] (2 responses)
Posted Jul 3, 2011 20:15 UTC (Sun)
by elanthis (guest, #6227)
[Link] (1 responses)
A further problem is a lack of conformance tests. This is the same flaw OpenGL suffers from, and is the primary reason why nobody in their right mind uses OpenGL unless they're forced to (e.g. because they're targetting non-Windows OSes, none of which have bothered to replace OpenGL with something not designed by amateur C hackers in the 1980's).
Posted Sep 28, 2016 14:10 UTC (Wed)
by Hi-Angel (guest, #110915)
[Link]
Posted Jul 4, 2011 13:23 UTC (Mon)
by rankincj (guest, #4865)
[Link] (1 responses)
Posted Jul 4, 2011 14:08 UTC (Mon)
by nix (subscriber, #2304)
[Link]
How to know if you are affected
How to know if you are affected
[ 0.000000] PCIe ASPM is forcedly enabled
[ 0.439490] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
How to know if you are affected
+{
+ if (!aspm_force)
+ aspm_clear_state = 1;
+}
How to know if you are affected
I wonder if this is a contributing factor to this bug I'm experiencing where the kernel's attempt to turn ASPM off on an e1000 device whose support is lethally buggy fails or is somehow reverted shortly after taking effect, leading to an interface freeze a few minutes to days later...
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
Note, there are some limiting factors on OEM hacks. If the OEM installs the 64-bit edition of Windows or if they use the 'Designed for Windows' logo then all drivers must pass a fairly challenging test suite (WHQL test). So there is some disincentive to modifying drivers. I'm not sure whether there is a similar test suite for the firmware.
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
same for everybody, not just us
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
PCIe, power management, and problematic BIOSes
So anyone who ran 2.6.37 OK can probably enable ASPM safely
So anyone who ran 2.6.37 OK can probably enable ASPM safely