A Plumber's Wish List for Linux

Posted Oct 7, 2011 18:32 UTC (Fri) by jra (subscriber, #55261)
Parent article: A Plumber's Wish List for Linux

Working POSIX aio, supported by the kernel (sigh).

Samba has needed this for 5+ years - works fine on Solaris. Don't see why I have to be a second class citizen on Linux.

to post comments

A Plumber's Wish List for Linux

Posted Oct 7, 2011 18:59 UTC (Fri) by aliguori (guest, #30636) [Link] (28 responses)

POSIX AIO is not a very nice interface. Signal based completions are awkward to use at best. In addition, aiocb only contains a flat buffer, not a scatter gather list. For something like QEMU, that makes it a useless interface since it can't support zero-copy I/O.

I'd rather see the effort put into making linux-aio actually work in a meaningful way with buffered I/O. linux-aio is at least a sane interface and is not restricted to the extremely slow process of POSIX standardization.

linux-aio has a proper file descriptor based completion mechanism and support for vectored I/O.

A Plumber's Wish List for Linux

Posted Oct 7, 2011 23:52 UTC (Fri) by jra (subscriber, #55261) [Link] (27 responses)

aliguori wrote:

> POSIX AIO is not a very nice interface. Signal based completions are
> awkward to use at best. In addition, aiocb only contains a flat buffer,
> not a scatter gather list. For something like QEMU, that makes it a
> useless interface since it can't support zero-copy I/O.

But guess what - it's a *STANDARD* that is supported on many other systems.

That's what makes it valuable to have working. We already have code that implements it.

I'm not saying *only* implement the standard. Do a nice Linux-specific interface underneath that the libc code then calls if you like, but don't ignore the standard interface as there's code out there that already uses it.

Of course it needs to work with buffered aio, that goes without saying :-).

A Plumber's Wish List for Linux

Posted Oct 8, 2011 16:44 UTC (Sat) by aliguori (guest, #30636) [Link] (24 responses)

You don't need kernel support for Posix aio. The reason it's so bad is that glibc serializes each request for reasons I don't understand. I tried to ask Uli but never got a response. We ended up reimplementing posix aio in QEMU and the performance is quite good. We've also added additional interfaces. The kernel would have to use a thread pool anyway unless every file system is written.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 0:35 UTC (Mon) by jra (subscriber, #55261) [Link] (1 responses)

I fixed the glib code to remove the serialization requirement and tested it. The results were interesting. For a micro benchmark given to me by a Samba OEM vendor more than doubled in speed, but then when testing inside Samba with the Intel NAS benchmark things went slower. Still not sure why. can you point me to your implementation in QEMU - maybe that will help.

A Plumber's Wish List for Linux

Posted Oct 30, 2011 11:21 UTC (Sun) by Lennie (subscriber, #49641) [Link]

I'm not familiar with the specifics but could something like that code be added to eglibc or an other project ?

Maybe as a separate API ?

If the performance really is improved that much it is silly not to be able to use it.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 0:46 UTC (Mon) by jra (subscriber, #55261) [Link] (20 responses)

Found the QEMU aio code. It appears to be llicensed GPLv2 *only*. WTF ??? Why would anyone do that ? Just to shaft more forward-looking GPL projects and force people to fork or re-implement ? Can you give me an explaination for this incredibly anti-social act, or if I'm wrong please show me the GPLv2+ text I've missed.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 11:39 UTC (Mon) by liljencrantz (guest, #28458) [Link] (14 responses)

Huh? Why is licensing your code under GPLv2 an «incredibly anti-social act»? Many people, including the Linux kernel developers have chosen GPLv2 as their preferred license.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 16:06 UTC (Mon) by jra (subscriber, #55261) [Link] (13 responses)

It's not licensing under GPLv2 that's anti-social, Samba also used to be licensed under GPLv2. It's licensing GPLv2-*ONLY* that's the anti-social part (and yes I've spoken to many Linux kernel developers about this).

The simple act of licensing GPLv2+ allows other projects that have moved to GPLv3 to re-use that code. It doesn't force the project that wants GPLv2 to move, it simply allows code reuse by other projects that have upgraded the license.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 16:50 UTC (Mon) by mpr22 (subscriber, #60784) [Link] (12 responses)

Releasing a work as GPLv2-or-later means effectively handing over control of the licence terms on the covered versions of the work to a third party in perpetuity; it's entirely reasonable to find that notion objectionable even if one regards the FSF as a trustworthy organization. Just because one happens to find the terms of GPLv2 suitable doesn't mean that one will find the terms of GPLv3 suitable.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 17:00 UTC (Mon) by jra (subscriber, #55261) [Link] (11 responses)

True, but forcing yourself to be "v2-only" isolates your project from the rest of the community. Now if that's what you want to do, then it's your choice, but it's not a social act, more a selfish one.

It also risks the continuance of your project if fatal flaws are found in your 'XX-only' license, as many believe to be the case with GPLv2. You might find your code used in ways you were specifically trying to avoid by your original choice of license.

Picking a 'XX-only' license is a touching faith in the bug-free nature of legal code. As programmers I would hope we were wiser than that.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 17:05 UTC (Mon) by mjg59 (subscriber, #23239) [Link] (5 responses)

Choosing any GPL license isolates you from a large part of the community. I think describing it as anti-social is rather extreme.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 17:45 UTC (Mon) by jra (subscriber, #55261) [Link] (4 responses)

I'm sorry, I didn't define my terms correctly - that's my fault.

When I use the word community, I'm considering the GPL-code-writing community - that's the group I feel a part of. There are other communities, and a broader FLOSS community using different licenses, but they're not ones that I would chose so I don't feel as much a part of them.

Within that narrower definition (although it is 70% of all Free Software out there) then chosing an XX-only license is an anti-social act, as it prevents wider reuse within GPL projects, which IMHO is the purpose of that license.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 18:41 UTC (Mon) by mjg59 (subscriber, #23239) [Link] (3 responses)

It's a community that's based on a large number of non-GPL components (plus one fairly significant GPLv2-only component). If you define "Community" in such a way that, say, X.org isn't part of the community, I think you're aligning yourself pretty differently to the majority of people contributing to free software development.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 21:20 UTC (Mon) by jra (subscriber, #55261) [Link] (2 responses)

Maybe so, but that's the community I feel the most affinity with (the GPL software producing community).

A Plumber's Wish List for Linux

Posted Oct 13, 2011 16:35 UTC (Thu) by bronson (subscriber, #4806) [Link]

'Antisocial' refers to society as a whole, not just the community you personally feel the most affinity with.

A Plumber's Wish List for Linux

Posted Oct 16, 2011 18:01 UTC (Sun) by obi (guest, #5784) [Link]

As it's supposed to be a community, you could simply ask for this specific code to be released GPLv2+.

Or even better, understand their apprehensions about not automatically trusting all future GPL versions, and ask for this code to be released v2 + v3 only.

If people would ask me to release some of my code under an additional or more liberal license so it can be reused I doubt I'd deny them.

So just ask in a friendly way.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 19:58 UTC (Mon) by fperrin (subscriber, #61941) [Link]

It also risks the continuance of your project if fatal flaws are found in your 'XX-only' license, as many believe to be the case with GPLv2. You might find your code used in ways you were specifically trying to avoid by your original choice of license.

If a fatal flaw in GPLv2 allows Stallman to come to your house and kill your kittens, then a GPLv2+ licensing enables you to release future versions of your software under GPLv3 where you are protected; but previous versions of your software are still available under the GPLv2 and Stallman can still get an old version and come slaughter your kittens (if wants to; in the GPLv2+ scenario, the choice of the exact license is down to the user).

A Plumber's Wish List for Linux

Posted Oct 10, 2011 21:59 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

by the same logic, choosing GPLv3+ isolates your project from parts of the community and so is anti-social

face it, when the FSF created the GPLv3 they split the community between those who use GPLv2 and those who use GPLv3, there are some who allow both, but in those cases the code can only move one way to GPLv3+ projects.

A Plumber's Wish List for Linux

Posted Oct 10, 2011 23:33 UTC (Mon) by jra (subscriber, #55261) [Link] (2 responses)

Chosing GPL at all is in itself a statement that you approve of that license and the copyleft provisions within. That in and of itself is a fairly strong statement in support of the FSF who are the creators and maintainers of the license.

Saying "I support GPL and copyleft, but I don't trust the people that created it" strikes me as a bit silly (and anti-social). It's the FSF that are moving it forward to deal with modern threats such as DRM-locked down hardware and software patents that simply didn't exist with earlier versions.

Anyway, this is getting distracting from the fact that the Linux kernel doesn't have properly working POSIX AIO, and the real technical discussions I'd prefer to be having, so I'm going to leave my politics at this comment, and get back to complaining that the kernel doesn't give me the free pony that *I* want :-).

A Plumber's Wish List for Linux

Posted Oct 11, 2011 0:14 UTC (Tue) by vonbrand (subscriber, #4458) [Link]

How is that silly? GPLv2 is out there, you can look it over and satisfy yourself that it does express what you want. GPLv3 doesn't, for a lot of copyleft advocates. And nobody can even hint at the changes GPLv4 might bring...

A Plumber's Wish List for Linux

Posted Oct 12, 2011 22:10 UTC (Wed) by Baylink (guest, #755) [Link]

> Saying "I support GPL and copyleft, but I don't trust the people that created it" strikes me as a bit silly (and anti-social)

I disagree with you, but I'm pretty certain you're not Hitler. :-)

I think it's equivalent to "Love your country, but never trust its government". That v2, which you could see, and choose, suited you, and you weren't sure later versions would not, declining to take advantage of the "or later version" language seems perfectly sane to me.

And gimme back my initials; it's confusing? ;-)

A Plumber's Wish List for Linux

Posted Oct 10, 2011 12:07 UTC (Mon) by mpr22 (subscriber, #60784) [Link]

The only licence choices that aren't "antisocial" by the definition apparently in use here are "CC-0" (i.e. as close to public domain as jurisdictions with certain inalienable creators' rights will let you get) and "new BSD".

Licenses?

Posted Oct 10, 2011 16:14 UTC (Mon) by vonbrand (subscriber, #4458) [Link] (3 responses)

Perhaps they just don't trust the FSF to respect the philosophy of the license they are comfortable with. The whole GPLv3 fracas showed clearly that this can be the case.

Licenses?

Posted Oct 10, 2011 23:39 UTC (Mon) by jra (subscriber, #55261) [Link] (2 responses)

I'm old enough to remember when the GPLv2 was beyond the pale and considered pure evil and communism (which is only a dirty word in the USA, and possibly China :-), no commercial company could *possibly* work with code under such a horrible and business-unfriendly license.

My, how times have changed. The same will happen with GPLv3 (probably after GPLv4 is announced :-)

If you don't trust the creators of the license, who do you trust to maintain it ? Do you think it doesn't need maintenance ? I know several lawyers in proprietary software companies that would disagree with you on that fact.

Licenses?

Posted Oct 11, 2011 0:19 UTC (Tue) by vonbrand (subscriber, #4458) [Link] (1 responses)

At least as far as I understand many Linux authors, they are fine with GPLv2 and don't agree with GPLv3, won't give up their copyright (or other rights, including the "change the license" right) to anybody within the Linux community, and definitely will never give the "change the license" right to somebody outside said group. Just remember the flamewars that erupted when it was suggested to move Linux to GPLv3.

Licenses?

Posted Oct 12, 2011 22:13 UTC (Wed) by Baylink (guest, #755) [Link]

And FWIW, there's nothing saying an author could not later relicense their own code as GPLv4, if v3 changes they didn't approve of were rolled back...

Strictly speaking, you're supposed to distribute the code under the version of the license it came with, but if a sole developer later relicensed his code under a newer license, and you took the old package from a 3rd party under that newer license, I'd be hard pressed to see anyone give you crap about it...

A Plumber's Wish List for Linux

Posted Oct 10, 2011 0:49 UTC (Mon) by jra (subscriber, #55261) [Link]

I actually did ask Ulrich and got a response (originally). He told me it was to throttle user space processes who might use too many threads. When I pointed out that the field inside the struct passed to aio_init() satisfied exactly this purpose, *then* he stopped responding to me :-).

A Plumber's Wish List for Linux

Posted Oct 8, 2011 17:48 UTC (Sat) by jcm (subscriber, #18262) [Link] (1 responses)

We have this wonderful delusion in the Linux community that we don't need no stinkin standards and things would be so much better if we just threw things away and did everything differently. And that's precisely what proprietary UNIX did for many years before efforts like POSIX.

Jon.

A Plumber's Wish List for Linux

Posted Oct 8, 2011 17:48 UTC (Sat) by jcm (subscriber, #18262) [Link]

Oh sorry, right, we're all Open Source so it doesn't matter. Puh-lease.