Weekly Edition
Return to the Kernel page
| |||||||||||||
The 3.11 merge window closes![[New logo]](/images/2013/3.11-logo.png){kind=logo}
Linus announced the release of 3.11-rc1 —
and the closing of the 3.11 merge window — on July 14. While the
merge window was open, 9,494 non-merge changesets were pulled into the
mainline kernel repository. The last
of those changes changed the kernel's codename to "Linux for Workgroups"
and modified the boot-time logo; the new version appears to the right.
Clearly, Linux development has moved into a new era.
Of those 9,494 changes, 1,219 were pulled since last week's summary. User-visible changes in that final batch of patches include:
Changes visible to kernel developers include:
Recent development cycles have lasted for about 70 days (though 3.10, at 63 days, was significantly shorter). If that pattern holds for this cycle, the 3.11 kernel can be expected around September 9. (Log in to post comments)
The 3.11 merge window closes Posted Jul 16, 2013 18:03 UTC (Tue) by yokem_55 (subscriber, #10498) [Link]
Given the timing of when this comes out and the expiration of Greg-KH's support of the 3.0.xx series, is it reasonable to infer the 3.11 will be the new long term stable release?
The 3.11 merge window closes Posted Jul 16, 2013 19:48 UTC (Tue) by smoogen (subscriber, #97) [Link]
I would say not until or unless Greg says so. He might decide that 3.9.x was his next or 3.10 or 3.12
The 3.11 merge window closes Posted Jul 16, 2013 20:10 UTC (Tue) by djc (subscriber, #56880) [Link]
Probably not 3.9:
The 3.11 merge window closes Posted Jul 18, 2013 15:31 UTC (Thu) by mgross (subscriber, #38112) [Link]
I hope its 3.10. If it ends up being 3.11 it will be very hard to move my product trees to 3.11 and harden the code before the end of the year.
The 3.11 merge window closes Posted Jul 16, 2013 18:44 UTC (Tue) by geofft (subscriber, #59789) [Link] I missed the definition of O_TMPFILE. The best explanation seems to be the previous LWN article:The new O_TMPFILE option to the open() and openat() system calls allows filesystems to optimize the creation of temporary files — files which need not be visible in the filesystem. When O_TMPFILE is present, the provided pathname is only used to locate the containing directory (and thus the filesystem where the temporary file should be). So, among other things, programs using O_TMPFILE should have fewer concerns about vulnerabilities resulting from symbolic link attacks.Pretty useful.
The 3.11 merge window closes Posted Jul 17, 2013 3:10 UTC (Wed) by proski (subscriber, #104) [Link] I hope the files would still be visible in the filesystem. Otherwise we would get DOS-style "hidden" and "system" files.
The 3.11 merge window closes Posted Jul 17, 2013 3:42 UTC (Wed) by geofft (subscriber, #59789) [Link]
This isn't DOS-style at all. If anything, this is UNIX-style hidden files. The semantics are exactly the same as if you picked a unique name, opened the file, and then deleted the file -- something you can do in UNIX, but not, traditionally, on DOS. (Windows NT attempts to support this, but doesn't do so well.) UNIX leaves files on disk until the last reference is gone, whether that reference is a name on disk or an open file handle.
The additional feature here is that there's no effort spent picking a unique name, just to stop using that name.
The 3.11 merge window closes Posted Jul 17, 2013 4:52 UTC (Wed) by viro (subscriber, #7872) [Link]
No. These files are created opened, without any links. So as soon as they are closed, they get deleted - just as a file you'd opened and unlinked. tmpfile() == fdopen(open("/tmp", O_TMPFILE | O_EXCL | O_RDWR, 0600), "rw");
_Without_ O_EXCL you are able to create a link to them later with something like linkat(AT_FDCWD, "/proc/self/fd/42", AT_FDCWD, pathname, AT_SYMLINK_FOLLOW) (if 42 is a descriptor of normal opened-and-unlinked file, you'd get -ENOENT from that, and so you would with O_TMPFILE|O_EXCL opened descriptors).
So there are two uses - one is race-free temp files (deleted when closed, never reachable from any directory, not subject to symlink attacks, not requiring to come up with unique names, etc. - basically, tmpfile(3) done right) and another is "create an initially unreachable file, write whatever you want into it, fchmod()/fchown()/fsetxattr() it as you wish, then atomically link it in, already fully set up".
The 3.11 merge window closes Posted Jul 17, 2013 6:15 UTC (Wed) by epa (subscriber, #39769) [Link]
In fact, that sounds like a better way to write files in general. No more pesky partially-written files if something goes wrong; instead open with O_TMPFILE and atomically link it with a certain name once you're finished.
The piece that's missing is an atomic link-and-unlink operation where you link a file into a directory with a given name, at the same time unlinking any file that was previously there with that name (or even renaming the existing file atomically).
The 3.11 merge window closes Posted Jul 17, 2013 6:16 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]
Also, an ability to create several files at once atomically would be appreciated.
The 3.11 merge window closes Posted Jul 17, 2013 9:27 UTC (Wed) by Karellen (subscriber, #67644) [Link]
"that sounds like a better way to write files in general. No more pesky partially-written files if something goes wrong;"
You can do that already by creating a file with the "wrong" name (e.g. "config.cfg.new") and calling rename(2) when you're finished. e.g.
rename("config.cfg.new", "config.cfg");
"instead open with O_TMPFILE and atomically link it with a certain name once you're finished."
Unfortunately, there currently is no way to create a new link to a file for which you only have a file handle.
There are some cases in which it wouldn't make sense, such as trying to link a socket fd into a filesystem, or creating a link in a mountpoint where the file does not reside, but a hypothetical fdlink() could return EXDEV as per rename() in that case. In any case, no-one's implemented it yet.
"The piece that's missing is an atomic link-and-unlink operation where you link a file into a directory with a given name, at the same time unlinking any file that was previously there with that name"
rename(2) already does this.
"(or even renaming the existing file atomically)."
You can do this with link(2), by e.g.
link("config.cfg", "config.cfg.old");
The 3.11 merge window closes Posted Jul 17, 2013 10:12 UTC (Wed) by epa (subscriber, #39769) [Link] Unfortunately, there currently is no way to create a new link to a file for which you only have a file handle.I thought the linkat trick described by Al Viro in the grandparent comment would achieve that.
I believe that However, it is atomic in the looser sense that the filename at any moment links to either the old file or the new one. That may be good enough for many applications.
The 3.11 merge window closes Posted Jul 17, 2013 10:40 UTC (Wed) by Karellen (subscriber, #67644) [Link]
"I thought the linkat trick described by Al Viro in the grandparent comment would achieve that."
Doh! That'll teach me to skim some messages.
Wow. That's incredibly neat. Hadn't thought of/seen that before. Thanks.
The 3.11 merge window closes Posted Jul 17, 2013 17:22 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
If the file doesn't currently have a filename, then there is no window where there are multiple paths to the file (not that having multiple paths to a file should be any problem, that's a 'normal' case on a Unix filesystem)
as long as the target path always exists, and always points at either the old or the new, you should be in good shape.
now, to be crash safe, you need to fsync the file before doing the rename, and you need to not be using ext3 which has such horrid fsync behavior.
The 3.11 merge window closes Posted Jul 17, 2013 10:43 UTC (Wed) by mjg59 (subscriber, #23239) [Link]
You've missed the bit where you have to fsync() the file because POSIX says the metadata can be updated before the data hits the disk and you can end up with an empty file, except you shouldn't do that because ext3 will then write out the entire journal and you'll block for several seconds.
The 3.11 merge window closes Posted Jul 17, 2013 23:35 UTC (Wed) by Karellen (subscriber, #67644) [Link]
"You've missed the bit where you have to fsync()..."
I've seen that argument before, but it's always confused me. Surely that's only wanted as protection against an unexpected system crash/failure? Except - I didn't think that POSIX made any guarantees at all in that event. I thought your OS was "allowed" to overwrite your partition tables and FS journals completely in the event of a crash and still be POSIX-compliant.
(If not, how does POSIX expect to guarantee otherwise, unless POSIX compliance requires the absence of certain classes of bugs?)
Looking at the rationale section of POSIX fsync[0] documentation, fsync() is allowed to be the null operation, or to not cause data to actually be written, and that fsync() correctness could be considered a QoI issue.
However, the Open Group website documentation the closest thing I have to the actual POSIX spec. If there is another section somewhere dealing with the general problem of compliance in the face of bugs/power outages which is more enlightening, I would welcome a link to it, or a quote from it.
(FWIW, I think that Linux writing metadata before data is a poor QoI decision, and that the filesystem devs should strive to do otherwise, no matter what POSIX allows. However, IANA Kernel/FS developer, and am not properly informed on how hard, impractical or pessimal that might be.)
[0] http://pubs.opengroup.org/onlinepubs/009695399/functions/...
The 3.11 merge window closes Posted Jul 18, 2013 0:25 UTC (Thu) by mjg59 (subscriber, #23239) [Link]
If we weren't concerned about such things, ext4 wouldn't default to committing every 5 seconds. We can't guarantee that systems will remain powered or that software is perfect, and so if your expectation is that your code will always either read the new file or the old file and never see an intermediate representation then you probably want to handle the not-uncommon case of the device ceasing to function at around the same time.
The meaning of fsync Posted Jul 20, 2013 17:56 UTC (Sat) by giraffedata (subscriber, #1954) [Link] I thought your OS was "allowed" to overwrite your partition tables and FS journals completely in the event of a crash and still be POSIX-compliant. The thing about adherence to any standard is that one specifies the very adherence with myriad conditions, most of them implied. So POSIX doesn't say, "if the system crashes, a read doesn't have to get back the same data that was written." Rather, the system designer says, "the system is POSIX-compliant as long as the system never crashes." And as I said, that condition is usually not actually spoken. There are tons of similar conditions: the superuser does not write directly to the disk; the disk drive never makes a mistake; cosmic rays don't change magnetic state; etc. Of course, designers do whatever they can to reduce the conditions; few systems today are offered on a "if the power ever goes out, nothing in the POSIX standard applies" basis. Fsync drives us into the awkward territory of robustness. Robustness is a system's ability to work when it is broken. That contradiction in terms is why any specification of fsync is bound to be fuzzy. It's like saying, "I will pay you back by Tuesday. If I don't, ..."
The 3.11 merge window closes Posted Jul 18, 2013 14:06 UTC (Thu) by Tobu (subscriber, #24111) [Link] The unavailability of good (O_PONIES) semantics continues to amaze me. The only option right now seems to be a combination of f(data)sync and deferred threads; but introducing threads has a nasty engineering cost. The last I've seen of these issues (on the XFS list), maintainers were willing to take a new flag (don't know if that's possible; the VFS seems misdesigned to ignore new flags, see O_TMPFILES above) or a new VFS syscall that might be progressively implemented.
The 3.11 merge window closes Posted Jul 18, 2013 14:23 UTC (Thu) by viro (subscriber, #7872) [Link]
FWIW, it's much worse than any VFS design flaws; those we could deal with, but there's fuck-all we can do about existing userland ABI. Which, for open(2), had been "ignore any unknown bits", all the way since the very beginning. And yes, it *is* a sucky ABI. Unknown values in flags, etc., should be rejected with "I don't know what are you talking about", rather than silently ignored. Sadly, open(2) is one of the places where that principle had been violated. What's really sad is that openat(2) did not add such validation, even though it had happened much more recently.
See the talk by Michael Kerrisk re ABI suckitude a while ago - this is a prime example of such ;-/
The 3.11 merge window closes Posted Jul 18, 2013 15:18 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]
I really wish that Linux would stop piling flags upon flags to existing syscalls. The man page for open() already looks like one of those 'interactive novells'.
Just create a new syscall, say open2(), with a better-designed ABI. Old programs can still use open() and new ones can use the new syscall to get new features.
The 3.11 merge window closes Posted Jul 18, 2013 19:58 UTC (Thu) by paulj (subscriber, #341) [Link]
And if a new syscall is added, at least divide the flag space up into "Don't care if not implemented" and "Mandatory, error if not implemented".
The 3.11 merge window closes Posted Jul 21, 2013 22:05 UTC (Sun) by nix (subscriber, #2304) [Link]
Oh, great. Because one of the things that's really *nice* about the Windows API is its profusion of Foo and FooEx and FooExEx calls.
No, what you do if you really think programs will care is introduce a new open2(), wire it to a new version of open() in glibc, change the values of all the O_* constants in glibc (but *not* the kernel) to some new value range that doesn't intersect the old, and have glibc redirect all calls using any old flag values to the old open() and all new ones to open(), mapping the 'new' flag values in the userspace API to the kernel values (probably by subtracting a constant). You can also expose the old flags under new names, OBS_EXCL and the like,. That way, old apps get the old syscall, new ones get the new syscall, and new apps that really, really want the old semantics can get them.
If you thought it mattered that much, and really needed to do it, that's how you'd do it. No uglifying programs with horrible open2() nonsense. (Yes, you need a new glibc version to use this, but you need a new glibc to use any new syscall *anyway*.)
Was this patch series included? Posted Jul 16, 2013 22:43 UTC (Tue) by roskegg (subscriber, #105) [Link]
https://git.kernel.org/linus/75485363ce8552698bfb9970d901...
Was this patch series to fix swap problems included in the 3.11 merge?
Was this patch series included? Posted Jul 16, 2013 22:48 UTC (Tue) by corbet (editor, #1) [Link] I'm not sure I understand the question...you gave a link to Linus's repository, so you're aware that the patch is in the mainline. Yes, it was pulled in during the 3.11 merge window, if that's the question.
Was this patch series included? Posted Jul 17, 2013 4:57 UTC (Wed) by roskegg (subscriber, #105) [Link]
Thank you for answering the question I meant to ask. In the early days of Linux, people weren't so gentle.
Was this patch series included? Posted Jul 17, 2013 17:04 UTC (Wed) by jimparis (subscriber, #38647) [Link] Given a git commit ID and a local up-to-date clone, you can figure that out with: $ git describe --contains 75485363ce8552698bfb9970d901f755d5713cca v3.11-rc1~99^2~392which tells you that the first tag containing this commit was v3.11-rc1, so it made the window. It used to be that you could get this information from kernel.org's "raw" gitweb output, saving the need for a local clone, but they recently switched to cgit which doesn't seem to provide that info.
Was this patch series included? Posted Jul 17, 2013 17:58 UTC (Wed) by mathstuf (subscriber, #69389) [Link]
Has X-Git-Tag been mentioned to cgit upstream?
Was this patch series included? Posted Jul 17, 2013 18:27 UTC (Wed) by jimparis (subscriber, #38647) [Link] I sent a mail to the kernel.org webmaster about a month ago, because they said "If you notice that something that used to work with gitweb no longer works for you with cgit, please drop us a note", but haven't received a reply. I haven't sent anything to cgit upstream though.
The 3.11 merge window closes Posted Jul 17, 2013 3:05 UTC (Wed) by timrichardson (subscriber, #72836) [Link]
Linux for Workgroups :) Although 3.1 was the first Windows for Workgroups release.
The 3.11 merge window closes Posted Jul 17, 2013 8:00 UTC (Wed) by rvfh (subscriber, #31018) [Link]
To most of us who were using computers at that time, _the_ WfW is definitely 3.11 (http://toastytech.com/guis/win311logo.gif).
The 3.11 merge window closes Posted Jul 17, 2013 8:16 UTC (Wed) by dsommers (subscriber, #55274) [Link]
That's fine for me, as long as the next release isn't Linux95 ... ;-)
The 3.11 merge window closes Posted Jul 17, 2013 9:04 UTC (Wed) by dgm (subscriber, #49227) [Link]
For that you'll have to wait for 4.0, codenamed "Cairo".
The 3.11 merge window closes Posted Jul 17, 2013 16:18 UTC (Wed) by drag (subscriber, #31333) [Link]
Coincidentally "Windows for Workgroups" was the first Windows operating system that Microsoft started using internally for development. Prior to that Windows and DOS systems were programmed using Microsoft's Unix variants.
I also expect that most people here already know this, but Microsoft's first operating systems were Unix variants, not DOS.
The 3.11 merge window closes Posted Jul 17, 2013 19:14 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]
Incorrect. UNIX has never been used much in MS (even during the Xenix era).
And by the time 3.11 was released, NT was almost done. And the NT group had been dogfooding their own builds for more than a year by then.
First Microsoft OS Posted Jul 17, 2013 22:59 UTC (Wed) by man_ls (guest, #15091) [Link] I also expect that most people here already know this, but Microsoft's first operating systems were Unix variants, not DOS.I found a detailed article which does not agree: apparently the first Xenix release was late in 1982, while DOS was released in August 1981. It is true that Microsoft had been working on Xenix since before they started with DOS, though. So it depends on how you define "first operating systems".
First Microsoft OS Posted Jul 18, 2013 13:33 UTC (Thu) by anselm (subscriber, #2796) [Link] When Microsoft got started, the BASIC interpreter on many computers doubled up as the operating system since it was the (ROM-based) program that was booted when the computer was switched on and supported commands for operating-system-like tasks. Since Microsoft was behind most BASIC interpreters of late-70s/early-80s vintage computers, that would make those Microsoft's first »operating systems«. {MS,PC}-DOS, when it was new, was basically a CP/M-80 knockoff for the 8088 CPU. It would probably have been reasonable at the time to do DOS development on CP/M, at least to the degree necessary to get a kernel running on the 8088 machine. It was possible to translate Z80 assembly language to 8088 assembly language automatically, so once you had a call-compatible CP/M-like kernel around you could probably get most of your other stuff across fairly quickly.
The 3.11 merge window closes Posted Jul 25, 2013 13:37 UTC (Thu) by Wol (guest, #4433) [Link]
Windows 3.1 did not contain any networking code.
WfW 3.11 was the first version to contain a TCP/IP stack, and the TCP/IP update disk for Windows 3.1 was released at the same time. It effectively upgraded 3.1 to 3.11.
I was there - the network stack (was designed to?) break WordPerfect 6, and I remember the grief it gave us, forcing us to choose between networking or word processing. That was a major factor in our office shifting from WordPerfect to Word.
Cheers,
The 3.11 merge window closes Posted Jul 25, 2013 20:19 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]
Windows and WfWg were two different products, both of which had 3.1 and 3.11 releases. But I think a lot of OEMs switched straight from Windows 3.1 to WfWg 3.11.
The 3.11 merge window closes Posted Jul 18, 2013 14:30 UTC (Thu) by Tobu (subscriber, #24111) [Link] Module loading behavior has been changed slightly in that the load will no longer fail in the presence of unknown module parameters. Instead, such parameters will be ignored after the issuing of a log message. This change allows system configurations to continue working after a module parameter is removed or when an older kernel is booted.That's short-sighted; it trades easy-to-diagnose failures for hard-to-diagnose failures. Udev rules won't be able to unambiguously require module features anymore. Surely the fact that previously working functionality now requires a human to read a logfile would be a red flag.
The 3.11 merge window closes Posted Jul 19, 2013 6:00 UTC (Fri) by rusty (✭ supporter ✭, #26) [Link]
modinfo will work for documented parameters, at least.
You were never supposed to remove module parameters, but that rule has proven difficult to enforce. This was chosen as a lesser evil, but of I'm wrong I will revert it before release.
Cheers,
module changes Posted Jul 19, 2013 16:37 UTC (Fri) by pflugstad (subscriber, #224) [Link]
A number of internet references[1] recommend disabling module (one-shot) by passing:
<moduleName>.blacklist=yes
on the kernel command line. I'm guessing this works because of the old behavior (fail module load due to unknown param). So this change would break that? Or is actual "blacklist" parameters handled differently?
[1]: http://askubuntu.com/questions/110341/how-to-blacklist-ke...
The 3.11 merge window closes Posted Jul 25, 2013 0:46 UTC (Thu) by heijo (guest, #88363) [Link]
What about applications that somehow accidentally used the O_TMPFILE flag value and worked correctly because the old kernels were ignoring it?
It seems to me adding any such flags breaks compatibility, and that adding a new "openat2" syscall is needed due to the preexisting mistake of not returning an error on invalid flags.
The 3.11 merge window closes Posted Jul 25, 2013 20:27 UTC (Thu) by BenHutchings (subscriber, #37955) [Link] O_TMPFILE now includes multiple bits that were already defined and could not be used together, plus one new bit (__O_TMPFILE). All of those bits must be set to create an unnamed temporary file. So, new userland that uses O_TMPFILE will fail cleanly on old kernels, and old userland that sets __O_TMPFILE will run unchanged on new kernels.
|
|||||||||||||
