LWN: Comments on "Trading off safety and performance in the kernel"

Trading off safety and performance in the kernel

javispedro — Tue, 20 Oct 2015 22:35:37 +0000

> I see that most laptops still come with spinning disk

At least in the country where I live, only 2 out of the 20 most sold laptops in Amazon contain a spinning disk. Those 2 peculiarly enough ship with Windows 7.

In fact, there are more laptops sold with eMMCs than spinning drives. I always learn something...

Trading off safety and performance in the kernel

mcortese — Fri, 11 Sep 2015 06:18:44 +0000

Now it's 2015, and, even within laptops, it's hard to find a system not shipping with connected standby

Funny. I see that most laptops still come with spinning disk: they shouldn't support Connected Standby as SSD is listed as a requirement. What am I missing?

Trading off safety and performance in the kernel

raven667 — Fri, 12 Jun 2015 17:50:04 +0000

That's just the link speed between the storage device and the main system and has little bearing on how fast you can actually write data to disk, especially if the disk is spinning rust. If the disk has to seek between writes then you aren't going to see more than around 100 writes per second which can be just a handful of megabytes, regardless of what the link speed is.

Your examples include a read heavy, write little web server and a db server which is probably explicitly flushing every IO to disk so that there is little data to write back in either case, neither of which is representative of how a laptop is used. It's easy to create a bunch of buffered writes, by copying a DVD image or compiling software or copying memory to disk for suspend, and on a laptop you may delay writes longer than normal to keep the disk subsystem in a low power state for as long as possible, leading to a storm of activity.

Trading off safety and performance in the kernel

bluefoxicy — Fri, 12 Jun 2015 17:32:59 +0000

> If the sync takes longer than 20 seconds, the suspend fails completely and the laptop stays on

SATA 3 6Gb/s: 15 gigabytes of recently-written dirty data to write

SATA 1 1.5Gb/s: 3.75 gigabytes

ATA100 100Mbit/s: 250 megabytes

I'm pretty sure this is a non-issue for any hardware made since 2003. /proc/meminfo shows 624kb of dirty pages on a big ass database server, 2712 on a busy Web server in a cluster. It's rare to have several gigabytes of unflushed disk just hanging around in memory; I've never seen more than a few megabytes.

Trading off safety and performance in the kernel

javispedro — Sat, 30 May 2015 16:34:47 +0000

Let's integrate by parts: "almost all of the devices sold these days" do not support S3, since currently the only non-negliglible architecture where S3 is supported is x86, and x86 is clearly a minority of "all devices sold these days", which includes ARM smartphones and tables, µC and other embedded devices.

The other part, "a huge amount of all 2015 laptops" do not support S3, is what should be apparent from the link I sent. The Microsoft logo requirements for Windows 8 laptops/tablets/hybrids recommend Connected Standby support. When Connected Standby support is present, the firmware _must disable_ S3 support; this is a Microsoft logo _requirement_ ( e.g. http://www.anandtech.com/show/8038/windows-81-x64-connect... ) and therefore every system out there which supports Connected Standby and Windows 8 _does not support S3_.

In 2014 the first systems with that started to ship (hybrids -- including the Surface Pro, the Dell Venue Pros, last Sony Vaios, the new Helix, etc.). _All_ hybrids and tablets from 2014 already shipped with Connected Standby, but only a few laptops. If you know of any hybrid/tablet that shipped in late 2014 without CS/AOAC please let me know.

Now it's 2015, and, even within laptops, it's hard to find a system not shipping with connected standby. The Dell XPS, the new Yoga and LaVie ThinkPads, the Asus Zenbooks all ship with Connected Standby.

However, it is a fact that there are some laptops left without Connected Standby, which (supposedly) still support S3. Which is why I said "a huge amount of all 2015 laptops", and not "all of 2015 laptops". Two examples: the Thinkpad X250 and the HP Spectre x360 do not support CS, to be best of my knowledge.

Hope this explains it.

Trading off safety and performance in the kernel

cesarb — Sat, 30 May 2015 16:08:35 +0000

> This very website: http://lwn.net/Articles/580451/

From that link: "They expect this mode to be used, as can be seen by the fact that machines that do not support the ACPI "S3" sleep state (a.k.a. suspend) at all will start shipping soon."

IMHO "machines [...] will start shipping soon" != "almost all of the devices sold these days (and a huge amount of all 2015 laptops)". So yeah, citation still needed.

Trading off safety and performance in the kernel

javispedro — Sat, 30 May 2015 14:43:01 +0000

This very website: http://lwn.net/Articles/580451/

Trading off safety and performance in the kernel

nix — Wed, 27 May 2015 22:31:54 +0000

This works great as long as you only have one generation of backups. More than that and bad things happen.

Trading off safety and performance in the kernel

toyotabedzrock — Tue, 26 May 2015 07:31:03 +0000

There is a simple solution to this that Microsoft uses. A file attribute that gets set when the file is modified. When a backup occurs that bit is reset.

Trading off safety and performance in the kernel

Cyberax — Mon, 25 May 2015 09:00:02 +0000

> That is a filesystems from decades ago. Yes it was broken, no question. Linux filesystems aren't like that. All non-trivial Linux filesystems do journalling of metadata, which is much safer than synchronous metadata updates. I cannot promise they are all 100% bug free in every release, but I am certain that calling 'sync' in the suspend path isn't going to usefully fix any bug that they might have.
As if on cue, today my laptop corrupted my filesystem during suspend/resume. I started synchronization of several large (~200G) directories with lots of small files from our local network and then totally forgot about it. Then I closed the laptop's lid and went home.

Resume failed (yet again) and after reboot my BTRFS filesystem refused to mount.

Trading off safety and performance in the kernel

neilbrown — Mon, 25 May 2015 06:52:10 +0000

> No, the file system is broken.

That makes no sense.

I agree that "If a filesystem needs the kernel to call sync before crashing then the filesystem in already broken" with the understanding that "needs" means "needs in order to protect the data that it is responsible for."
Data that has not yet been written to the filesystem is certainly not the filesystem's reponsibility.
Data that has been written but hasn't been the subject of 'fsync' is also not completely the filesystem's responsibility (unless you mount with '-o sync').

> one example of a broken file system losing data of a popular application

That is a filesystems from decades ago. Yes it was broken, no question. Linux filesystems aren't like that. All non-trivial Linux filesystems do journalling of metadata, which is much safer than synchronous metadata updates. I cannot promise they are all 100% bug free in every release, but I am certain that calling 'sync' in the suspend path isn't going to usefully fix any bug that they might have.

It also sounds like that "popular application", which was emacs, wasn't calling 'fsync' as it should and as it certainly now does.
Yes - bugs should be fixed. But let's not scatter "sys_sync" calls around and pretend that fixes them.

Trading off safety and performance in the kernel

simoncion — Sun, 24 May 2015 06:50:02 +0000

> In fact, almost all of the devices sold these days (and a huge amount of all 2015 laptops) do not even have the ability to enter S3 state or anything that would remotely resemble that.

Citation *seriously* needed. S3 is Suspend To Ram. Page 12 of this 2014 Intel datasheet [0] indicates that a large number of mobile i3, i5, and i7 processors support S0, S3, S4, and S5 mode. (Check page 10 for the list of processors.)

[0] http://www.intel.com/content/dam/www/public/us/en/documen...

Trading off safety and performance in the kernel

anton — Sat, 23 May 2015 16:20:31 +0000

If userspace needs the kernel to call sync before crashing then the user-space is already broken.

No, the file system is broken.

Are there actually any non-trivial apps which don't save their data properly?

No, there are just file systems (e.g., ext4) which do not provide decent guarantees and use this kind of rethoric to justify their poor behaviour. I expect that pretty much all non-trivial applications do not jump all the time through all the hoops that some developers of file systems expect of them; that's because they have no good way to test that they meet the expectations of these file system developers, and most application developers probably have many more urgent things to care about.

Anyway, one example of a broken file system losing data of a popular application (including the autosave files that the application produces regularly) is here.

Trading off safety and performance in the kernel

scientes — Fri, 22 May 2015 05:50:27 +0000

The backup program example still applies to chunks however. How about having mtime still update every 24-hours if the file is modified (like how noatime currently works)?

Trading off safety and performance in the kernel

Wol — Thu, 21 May 2015 21:58:52 +0000

> Are there actually any non-trivial apps which don't save their data properly?

Okay, it's not linux, but ... MS Word ?

(maybe it's changed, but I OFTEN lose data if I'm working on a document and it crashes - often it's the attempted auto-save that causes the crash :-(

Cheers,
Wol

Trading off safety and performance in the kernel

mathstuf — Mon, 18 May 2015 19:32:52 +0000

> There are genuine reasons why so many people carry their Windows or Linux laptop open around the office despite safety rules (not Macs interestingly enough).

Am I the only person who disables suspend-on-lid-close? For all OS variants. Never did like that behavior…

Trading off safety and performance in the kernel

neilbrown — Sun, 17 May 2015 04:21:08 +0000

> Yes, the "sync" semantics are "wait for any pending writeout to complete, then start writeout on any dirty data",

Actually, I'll have to backtrack on this. I can find no evidence in historical Unix, all the way up to 4.3BSD, to suggest that the 'sync' system call would wait. It just initiated IO. So maybe calling it 3 times makes sense.

Linux (roughly) followed that approach until Linux 1.3.20. That version introduced the change:

--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -228,7 +228,7 @@ int fsync_dev(dev_t dev)
 
 asmlinkage int sys_sync(void)
 {
-       sync_dev(0);
+       fsync_dev(0);
        return 0;
 }

To give the fuller context:

void sync_dev(dev_t dev)
{
        sync_buffers(dev, 0);
        sync_supers(dev);
        sync_inodes(dev);
        sync_buffers(dev, 0);
}

int fsync_dev(dev_t dev)
{
        sync_buffers(dev, 0);
        sync_supers(dev);
        sync_inodes(dev);
        return sync_buffers(dev, 1);
}

asmlinkage int sys_sync(void)
{
        fsync_dev(0);
        return 0;
}

The second arg to sync_buffers() says whether it should 'wait'. So fsync_dev() waits, sync_dev() doesn't.

Exactly what is waited for when is hard to track. I'd need to read a lot more code to see what things sys_sync waits for today. But it does wait for something, but before 1.3.20, and all through "Unix", it certainly didn't wait for everything.

Trading off safety and performance in the kernel

neilbrown — Sat, 16 May 2015 20:42:32 +0000

Yes, the "sync" semantics are "wait for any pending writeout to complete, then start writeout on any dirty data", so 2 is sensible and 3 is superstitious.

http://pubs.opengroup.org/onlinepubs/7908799/xsh/sync.html

This is part of why calling sys_sync() once in the suspend path is wrong (twice has been suggested), though I'm not certain if the Linux implementation exactly matches the specification.

Trading off safety and performance in the kernel

dlang — Sat, 16 May 2015 20:14:42 +0000

From what I understand, the issue was that a single sync could return to the command line before the data was actually on disk, but a second one couldn't start until the first finished.

that still doesn't justify three invocations, but would justify two.

Trading off safety and performance in the kernel

ghane — Sat, 16 May 2015 15:52:18 +0000

neilbrown wrote:
> But the point isn't really how long it takes. The point is that the 'sync' call really doesn't belong there. The benefit it provides is much more superstitious than scientific.

Back on the late 80s, I was taught to type:
sync ; sync ; sync

before a shutdown or reboot. I assume this was so that the SVR5 kernel would know I really wanted to sync.

Trading off safety and performance in the kernel

faramir — Sat, 16 May 2015 11:54:00 +0000

My Dell Lattitude D530 fails to resume about 1/3 of the time. I suspect a software bug as I think it worked back in the days of Ubuntu 10.04. I keep hoping some random software update will fix the problem.

Trading off safety and performance in the kernel

marcH — Fri, 15 May 2015 20:35:56 +0000

> In my case it's company policy that VPN sessions don't survive a suspend.

Well I feel sorry for you but that's not our case; suspend issues are why they do it here.

Trading off safety and performance in the kernel

bronson — Fri, 15 May 2015 19:55:08 +0000

Me too. That was surreal.

Trading off safety and performance in the kernel

tialaramex — Fri, 15 May 2015 18:23:47 +0000

> There are genuine reasons why so many people carry their Windows or Linux laptop open around the office despite safety rules (not Macs interestingly enough).

In my case it's company policy that VPN sessions don't survive a suspend. When you open the laptop back up, even after 30 seconds, the VPN client reminds you of the policy and prompts you to start over. You need to find your RSA dongle, go through the authentication again, reconnect, you'll get a new IP address and so of course all existing connections are dropped.

My company is a complete disaster for IT policy, but then, so are thousands of other large employers around the world. So this is a real scenario, even though it's an unnecessary and obnoxious one.

Trading off safety and performance in the kernel

flussence — Fri, 15 May 2015 17:27:39 +0000

I'm aware of that, but there's a huge amount of software in the wild convinced that it is: https://bugs.gentoo.org/show_bug.cgi?id=nonbash

No battery [was: Trading off safety and performance in the kernel]

bnorris — Fri, 15 May 2015 06:12:57 +0000

I saw that, but it should also be at least mentioned in Documentation/ABI/testing/sysfs-power alongside all the other /sys/power/disk options. Maybe I'll send a patch, if you don't.

Trading off safety and performance in the kernel

marcH — Fri, 15 May 2015 05:19:36 +0000

> You seem to be saying that it can't take enough time to bother you, and I suspect you are correct. Len Brown, by submitting the patch, is saying that it *does* take enough time to bother him.

Whatever two particular individuals experience does not matter much, can we please go back to statistics?

> Are you saying he is wrong?

I really wonder where does that come from... are you related maybe? :-)

> Undoubtedly true. This has no effect on whether placing a 'sys_sync' at that point in the code actually provides any benefit. At all.

The connection is: reliability is inversely related to complexity and users simply want their data saved before their computer susp...crashes. See various horror stories in the other comments.

> Laying aside for the moment that 1% of 1% is 0.01%,

Agreed! Let's also lay aside that 0.1% of 1% is 0.001%. And a few others?!

> these numbers are meaningless.

They're semi-random examples, but not completely meaningless. The very simple point I was trying to drive (and hoping not to have to detail) is just: the kernel is never going to please every single use case for every single user. Proof: there are practically no hardware device shipping with an totally unpatched mainline kernel. The mainline only has code that has a significant number of actual users. So I think we all agree it's all about how [un]common is this or that use case. Statistics and trade-offs.

> So removing the the sys_sync call from suspend in the kernel is already only going to affect few of the users that your are probably thinking of as the 99.99%.
> But one of the drivers for this change is, apparently, android.

Thanks for the info and also the reminder; I got distracted by the laptop stories filling almost the entire comments space.

Trading off safety and performance in the kernel

mjg59 — Fri, 15 May 2015 05:04:55 +0000

> The example given is that a machine running at max speed for 10 min will overheat while one running for several hours at the low speed will not.

Uh? I'm possibly missing something here, but I don't see any references to that example.

Trading off safety and performance in the kernel

mjg59 — Fri, 15 May 2015 04:51:36 +0000

Oh, right. Yes.

> But keep in mind that I'm not saying that reducing the clock speed is always the right thing to do, I am just unconvinced that it's never the right thing to do the way that you seem to be.

From https://lwn.net/Articles/644541/ (written by you)

> If you are saving the contents of RAM to disk, then you aren't going to finish any sooner at 5GHz clock than at 500MHz, the limiting factor is going to be your disk I/O performance. So if you can do this at 500MHz rather than 5GHz, you generate significantly less heat.

From https://lwn.net/Articles/644549/ (written by me)

> I'm not saying that it's impossible, but it's certainly not obvious.

…

Trading off safety and performance in the kernel

marcH — Fri, 15 May 2015 04:46:50 +0000

> But keep in mind that I'm not saying that [X] is always the right thing to do, I am just unconvinced that it's never the right thing to do the way that you seem to be.

I had to waste 5 minutes reading the entire thread again to make sure I did not dream and that the exact opposite happened.

Trading off safety and performance in the kernel

marcH — Fri, 15 May 2015 04:33:55 +0000

CDDB is not 100% reliable but it's useful and popular anyway.

At the start it was very incomplete yet it became popular very quickly. Same as many other crowd-sourced services.

No battery [was: Trading off safety and performance in the kernel]

bojan — Fri, 15 May 2015 00:48:38 +0000

https://www.kernel.org/doc/Documentation/power/swsusp.txt

No battery [was: Trading off safety and performance in the kernel]

bnorris — Fri, 15 May 2015 00:07:37 +0000

Thanks for the tip. BTW, the "suspend" option is not documented in Documentation/ABI/testing/sysfs-power.

Trading off safety and performance in the kernel

mjg59 — Thu, 14 May 2015 23:56:15 +0000

Shallow C states are basically free on modern CPUs. Deeper ones will drop cache, but that's basically irrelevant in the case we're discussing.

Trading off safety and performance in the kernel

dlang — Thu, 14 May 2015 23:47:52 +0000

That's not the issue. The example given is that a machine running at max speed for 10 min will overheat while one running for several hours at the low speed will not.

for this example, race to idle fails if it takes too long because the system will overheat, while running at a lower speed, even if it takes a lot more time and power will succeed and not damage things.

race-to-idle requires a very specific combination of power/performance at the different states (full speed, partial speed, and idle). That combination has not always been the case and there's no reason to believe that it is going to continue to always be the case. Idle does not always mean that it requires zero power (even for the component that's idled, let alone for the entire system)

Trading off safety and performance in the kernel

dlang — Thu, 14 May 2015 23:42:43 +0000

>> and do the shallow C states really save that much power over a lower clock speed?

> Yes. Even the most shallow C state will unclock the core, and running at 0MHz is somewhat cheaper than running at 500MHz.

remember that switching C states isn't free (in either energy or time), so it may not be a win if you don't stay there very long.

We obviously have very different expectations in how the hardware is going to behave at the different states. But keep in mind that I'm not saying that reducing the clock speed is always the right thing to do, I am just unconvinced that it's never the right thing to do the way that you seem to be.

Trading off safety and performance in the kernel

dlang — Thu, 14 May 2015 23:34:28 +0000

good luck. How are you going to reliably identify what system you are running on? With some vendors even the exact model number isn't enough because they change the internals without changing the model number

Trading off safety and performance in the kernel

neilbrown — Thu, 14 May 2015 22:31:23 +0000

> "Don't break userspace" - even userspace bugs.

If userspace needs the kernel to call sync before crashing then the user-space is already broken. Systems can crash without entering suspend first.

But that is a big "if". Are there actually any non-trivial apps which don't save their data properly?

Trading off safety and performance in the kernel

neilbrown — Thu, 14 May 2015 22:27:29 +0000

> Yes it is, otherwise this entire discussion would not even exist.

I'll be more precise. The point isn't the amount of time it takes, it is the fact that it takes any time at all.

You seem to be saying that it can't take enough time to bother you, and I suspect you are correct. Len Brown, by submitting the patch, is saying that it *does* take enough time to bother him. Are you saying he is wrong?

> Power management has always been complex and does not look like it's getting simpler any time soon.

Undoubtedly true. This has no effect on whether placing a 'sys_sync' at that point in the code actually provides any benefit. At all.

> Why bother the change and disruption if it's only for 0.001%.

Laying aside for the moment that 1% of 1% is 0.01%, these numbers are meaningless.

The vast majority of users get sync called (at least) twice on suspend - once by some user-space tooling and once by the kernel. Most (possibly all) distros already do this.
So removing the the sys_sync call from suspend in the kernel is already only going to affect few of the users that your are probably thinking of as the 99.99%.

But one of the drivers for this change is, apparently, android. I think the user-base there is a little more than 0.001%.

Trading off safety and performance in the kernel

mjg59 — Thu, 14 May 2015 21:42:16 +0000

If the system is in any kind of state where it has an effectively unbounded amount of work to perform then the situation changes pretty significantly. There are various cases where apps behave badly when they lose network connectivity and spin trying to reconnect, for instance.