The kernel developer panel at LPC
The initial topic was entry into kernel development, and the panelists' experience in particular. Khan, who got started around seven years ago, said that her early experience was quite positive; she named Tim Bird as a developer who gave her a lot of good advice at the beginning. Abbott started by tracking down a bug that was causing trouble internally; after getting some feedback, she was able to get that work merged into the mainline — an exciting event. Schumaker started with a relatively easy project at work. Lawall, instead, started by creating the Coccinelle project back around 2004. Her experience was initially somewhat painful, since the patches she was creating had to go through a lot of different maintainers.
It had been a busy week at LPC, Stewart said, asking the panelists what stood out for them. Khan called out the networking track as a place where she learned a lot, but also said that the conference helped her to catch up with what is going on with the kernel as a whole, which is not an easy thing to do. She mentioned the sessions on the kernel's code of conduct and the creation of a maintainer's handbook.
![[The panel]](https://static.lwn.net/images/conf/2018/lpc/kernel-panel-sm.jpg "Shuah Khan, Anna-Maria Gleixner, Laura Abbott, Anna Schumaker, Julia Lawall, Kate Stewart")
Gleixner said that, as a relative newcomer to the kernel, she found it helpful to hop around between tracks. She did follow the realtime microconference in particular, though, and described it as fun. Abbott enjoyed the hallway track (discussions outside of any organized session), along with the WireGuard and GitLab talks and the Android microconference. Schumaker mostly stuck with the Kernel Summit track, and agreed with Khan about code of conduct and maintainer's handbook sessions. Lawall called out the testing and fuzzing microconference and the realtime microconference.
Safety-critical, realtime, and more
Switching subjects, Stewart said that interest in using Linux in safety-critical application is growing; she asked the panelists where they thought the biggest gaps were in this area. Abbott replied that it mostly comes down to testing. The old claim that "many eyes" can find bugs is true, but it's better if those eyes are supplemented with testing; the good news is that the kernel is finally reaching a point where it has a good set of automated tests. Lawall agreed with that assessment, and suggested that we could improve in the area of static-analysis tools as well. We have some of these tools and they overlap coverage in various ways; it would be good to bring them together in a more coherent way. Khan also agreed, noting that the fuzz testing being done by the Syzbot project has been especially helpful, and that the unit testing framework in the kernel has now grown to test 46 different subsystems.
A related subject is the realtime patch set which, we have been informed, should be fully merged within a year. Stewart asked: what will change when that happens? Gleixner responded that realtime will not be different from any other kernel feature; if you break it, you'll have to fix it. A likely problem spot is code that disables preemption, creating unwanted latencies; we need better test coverage, she said, to find places where that is done, especially since it's not possible to test all drivers currently.
Since testing seemed to be on everybody's mind, Stewart asked what the plans were to improve testing in the future. Khan said that the kselftest framework remained weak, especially when it comes to driver tests, and that contributions would be welcome. The PowerPC and x86 architectures have reasonable test coverage, but Arm could use some help.
Gleixner said that there are some tests for realtime behavior, but the coverage is still small. The primary focus has been on detecting changes in response times. Abbott said that she is focused on the subsystems that tend to break when users get their hands on them; that means graphics and input drivers, for example. Third-party modules can also be a problem and could benefit from better test coverage.
Lawall, unsurprisingly, is working on adding more features to Coccinelle. Ensuring that initialization and exit sections are properly annotated is one area of interest. There is also work being done on a tool that can examine a handful of example patches and create a semantic patch that will effect similar changes elsewhere in the tree — an idea that drew enthusiastic applause from the audience. She warned that the result may never be perfect, but it can hopefully serve as a good starting point. A related project is a tool to detect code that could create problems when being backported to older kernels.
What changes will the new classes of hardware vulnerabilities bring to our processes? Khan said that she learned a lot from having to make a number of fixes to one of her drivers; it is good to think about where things could go wrong and to look for potential issues when reviewing code. Some of the proof-of-concept exploit code for these vulnerabilities is finding its way into the self-test framework to help protect against regressions in the future.
Abbott noted that the embargoes around these vulnerabilities have been "touchy", but they are needed for coordination between groups. She is one of the developers in the front line behind the security@kernel.org alias, but tries to handle things in the open whenever possible. The closed nature of the response to Meltdown and Spectre was a big problem, she said.
Tools and documentation
There was a brief discussion on whether the kernel community has the right tools to support long-term kernels. As Abbott noted, this work requires a good view of what is in any given kernel tree to be able to tell if any given patch should be backported to it. Khan admitted that she doesn't always think to send relevant patches to the stable team, so any tools that can help in that regard are useful. She mentioned cregit as a valuable tool. Schumaker added that the machine-learning work being done to identify candidates for stable backports has also been helpful.
Maintainers are a limited resource in almost every project; what can the kernel do to improve the situation there? And, Stewart asked, does the kernel's email-based process still work? Khan is looking forward to the upcoming maintainer's handbook as one helpful development in that area. Abbott said that working with email does present a bit of a learning curve; anybody can learn how to be effective with it, but it's worth asking whether email is really still the best way to collaborate. It works well for maintainers, she said, but perhaps less well for contributors.
A member of the audience noted, to applause, that perhaps the process is not broken even if today's kids prefer web pages. Stewart noted that there are over 60,000 files in the kernel and a lot of developers working with them; that represents a great diversity of opinions that can be mined for better ways to work. Khan said that the community has been continuously evolving and will continue to do so. Lawall added that documenting the details of how subsystems work, as the maintainer's handbook is expected to do, will help a lot; the differences between subsystems can be frustrating for developers now.
James Bottomley asked the panel for one thing that they would change at the Linux Plumbers Conference to make it more welcoming for new attendees; Abbott responded that Bottomley should be required to personally introduce himself to every one of them. More seriously, she said that it would be good to have a way to indicate that she is truly enthusiastic about speaking to new developers. Khan said that the green-dot stickers provided at the conference — applied to an attendee's badge to indicate a willingness to be approached — do work, and that the hallway track had been awesome.
Documentation returned to the fore as a member of the audience asked who should be writing kernel documentation. Gleixner said there is no easy answer to that question; if she documents something that is clear to her, she is likely to miss points that others need. Lawall said that the code itself should be the documentation. That includes good commit messages to explain why changes have been made; that is something the kernel community generally does well now. Schumaker said that reviewers should be asking about documentation.
Which is worse: out-of-date documentation, or documentation that's missing entirely? Gleixner voted for the former, since it creates confusion. Abbott said it depends on the nature of the documentation; some of it exists, for example, to explain the design decisions that were made and will remain useful even as the code evolves.
The final questions had to do with scaling the kernel community. The kernel is one of the fastest-moving projects now, but what would it take to get to a project that is ten times bigger? Khan said that it would simply be impossible to keep up with such a project. Lawall noted that, as the code base gets bigger, making progress becomes harder. Making changes gets more painful, so people just don't bother. The solution, of course, is better tools; Abbott, too, said that more automation will be required for the community to scale successfully.
[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting my
travel to LPC.]
| Index entries for this article | |
|---|---|
| Conference | Linux Plumbers Conference/2018 |
Posted Nov 19, 2018 20:04 UTC (Mon)
by marcH (subscriber, #57642)
[Link] (12 responses)
"First they ignore you, then they laugh at you,..."
Posted Nov 22, 2018 7:26 UTC (Thu)
by nhippi (subscriber, #34640)
[Link] (11 responses)
http://lists.infradead.org/pipermail/linux-arm-kernel/201...
Posted Nov 22, 2018 11:25 UTC (Thu)
by mageta (subscriber, #89696)
[Link] (1 responses)
Posted Nov 22, 2018 15:45 UTC (Thu)
by marcH (subscriber, #57642)
[Link]
gmail offers POP and IMAP
When I use Gerrit I use its web interface AND its command line AND its emails AND git - whichever is the more efficient for any particular task.
Longer discussion if you're interested at https://lwn.net/Articles/768483/
Posted Nov 23, 2018 12:28 UTC (Fri)
by kdave (subscriber, #44472)
[Link] (2 responses)
80 columns of code can fit to 1 monitor width 2-3 times, without trimming lines. This is quite important when comparing code side-by-side, either with previous revision or when resolving conflicts using e.g 'git mergetool -t vimdiff'. If this is seen as 'hazing' and pestering, please try to understand the impact of such code on those who has to maintain it and read it over and over again under stress, looking for bugs and doing reviews. Formatting properly gives the same look and feel and unless it's not done consistently, there's no point doing it at all. Removing distractions allows to focus on the actual problems in the code.
Posted Nov 27, 2018 7:39 UTC (Tue)
by marcH (subscriber, #57642)
[Link]
Posted Dec 4, 2018 15:24 UTC (Tue)
by nix (subscriber, #2304)
[Link]
(After all, the requirement from typography is vaguely-70-chars at handheld book distances with usual book fonts -- there is nothing really like multiple indentation levels there, so making it a distance from the left margin is in effect picking a much *smaller* limit for a lot of code, which is known to be *harder* to read. Where typography does have requirements for indented lists, usually the indentation is on *both* sides to make the list clearer visually. I've never seen a coding style with *that* requirement.)
Posted Nov 29, 2018 19:59 UTC (Thu)
by jschrod (subscriber, #1646)
[Link] (5 responses)
It's a pity that decades of research about readability (actually, hundreds of years, if you include typography) is termed _archaic_ -- but it's telling, too.
Restricting one line's width is not an implication of old-style terminals with 80 columns -- it's the other way round. Terminals got that width *because* that 72-character line length was a de-facto standard already.
And yes, I'm old enough to have worked with those terminals; actually I started to program with punch cards. Those *really* had the 72-80 line width limitation. (80 only if one didn't use ISAM.)
Posted Nov 30, 2018 19:17 UTC (Fri)
by jezuch (subscriber, #52988)
[Link] (4 responses)
Posted Nov 30, 2018 19:32 UTC (Fri)
by mpr22 (subscriber, #60784)
[Link] (1 responses)
I certainly use a wider field than 80 columns to do it, but I try to keep each line's non-indentation width down to around that kind of mark or narrower.
Posted Dec 1, 2018 11:58 UTC (Sat)
by jezuch (subscriber, #52988)
[Link]
Joking aside, there obviously needs to be some kind of limit. I usually set it at 120 these times, as 80 is too constricting most of the time (I work with Java). I think the Rust community is much more accepting in this regard. They indent like crazy, given their rules on aligning dots on iterators ;)
Posted Dec 1, 2018 13:43 UTC (Sat)
by jschrod (subscriber, #1646)
[Link] (1 responses)
One can argue that the width of the filled part of a line is only relevant. Within object oriented languages where methods tend to have a basic indentation of 8 characters for their code, this would push the acceptable line width from 80 to 90. That makes roughly no difference.
Of course, if one has regularly indentation levels of 4 or 5 units, then the remaining horizontal space starts to be short. But IMNSHO, that's unreadable code anyhow and should be decomposed.
Posted Dec 1, 2018 14:35 UTC (Sat)
by jezuch (subscriber, #52988)
[Link]
Posted Nov 19, 2018 20:42 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (22 responses)
Posted Nov 20, 2018 10:31 UTC (Tue)
by lkundrak (subscriber, #43452)
[Link] (1 responses)
Oh yes, this. Good maintainers take great care to attribute changes properly, acknowledging that it often is the only thing that the contributor gets in exchange for their work. Even if they're not satisfied with the patch and end up solving things differently.
That said, one quickly learns that a good idea to set one's ego aside when contributing to free software projects, and not insist on leaving footprints in the concrete. The act of reviewing one's patch is in itself an act of kindness, even when the surrounding communication is not ideal. I think it tends to be less fun than writing code or fixing things and the maintainers still do so on a daily basis.
Posted Nov 20, 2018 15:34 UTC (Tue)
by mathstuf (subscriber, #69389)
[Link]
Interestingly, I've also run into projects which are eager to add me to a contributor list for a simple typo fix. I see it as a kind of excess, but hey, whatever :) .
Posted Nov 20, 2018 16:59 UTC (Tue)
by marcH (subscriber, #57642)
[Link] (18 responses)
I bet it was a combination of:
It's pretty common to submit some idea to some higher authority' and see it rejected at first. Later we find this actually planted a seed (among other seeds) and the higher authority genuinely thinks the idea comes from him or her.
Posted Nov 26, 2018 15:58 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (17 responses)
Posted Nov 26, 2018 20:33 UTC (Mon)
by nix (subscriber, #2304)
[Link] (15 responses)
Posted Nov 26, 2018 21:13 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (14 responses)
Posted Nov 26, 2018 21:31 UTC (Mon)
by karkhaz (subscriber, #99844)
[Link] (7 responses)
In modern C, they are not equivalent because expression-1 can be a declaration. If expression-1 is a declaration in the first statement, then the scope of the variable is the loop body only. In the second statement, expression-1's scope is the scope enclosing the loop. Not the same thing. Therefore there is a use for for-loops, when you want to initialize a variable just once but have its scope be only the body of the loop. With a while-loop you would have to resort to this: with the extra brackets at the beginning and end to make
Posted Nov 26, 2018 21:54 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (6 responses)
You like for-loops. Hence, you find uses for them. I think they're an example of a too-low-level abstraction for the reasons I gave. Hence, unless I encounter some real code which would actually benefit from them, I'm not going to use them.
Posted Nov 26, 2018 22:59 UTC (Mon)
by nix (subscriber, #2304)
[Link] (5 responses)
The first arg to for is demonstrably *not* useless, because it introduces a variable scope which is otherwise unavailable. That you consider this a "C++ misfeature" does not negate the fact that it *has a use*. It's just a use that you personally happen not to like.
Posted Nov 26, 2018 23:17 UTC (Mon)
by nix (subscriber, #2304)
[Link] (3 responses)
In particular, note the first chunk of code in <https://lwn.net/Articles/628151/>. (However, this thread is massively saner than the slow-moving hilariously terrible trainwreck which was the coding style being discussed there.)
Posted Nov 27, 2018 0:27 UTC (Tue)
by karkhaz (subscriber, #99844)
[Link] (2 responses)
Yes, I do remember that thread! What a breathtaking spectacle of eye-wateringly deranged code that was. I kept waiting for somebody to reveal that the thread was an elaborate prank, or even just a wayward acid trip, but it didn't stop getting worse. The most surreal aspect was the deadpan, straight-faced insistence---maintained unwaveringly through the entire thread---that the code was clearer and less error-prone, and that unyielding adherence to delinquent style principles is more important than respect and consideration for the reader's intuition. The thing is, there were a few half-interesting ideas in that codebase. Stuff like calling a function like this: i.e. writing the parameter names next to the arguments...so that fragment is pretty ugly, but I've seen other languages that have cool sugar to make this work. In OCaml, you can write the parameter names prefixed by a tilde: (and you can reorder the arguments as I did, since they have the parameter names attached). If you make the effort to make the names of the variables you pass as arguments the same as the parameter names, you can omit the colon and the name of the argument (i.e.
Posted Nov 27, 2018 1:42 UTC (Tue)
by anselm (subscriber, #2796)
[Link]
Python lets you say, effectively,
which is arguably even more intuitive than what OCaML does.
OTOH, in Python for is not simply syntactic sugar for while, and Python's for and while loops have else, which is sometimes useful, as in
Finally, back when I was still programming in C I mostly used to think of for loops as “counting” through the elements of an array, a linked list, etc. and of while loops as “open-ended”. Of course we all know that from the language definition that distinction doesn't really exist, but when used with forethought it can help make programs more readable by playing to that intutition.
Posted Dec 2, 2018 12:02 UTC (Sun)
by paulj (subscriber, #341)
[Link]
Posted Nov 26, 2018 23:21 UTC (Mon)
by neilbrown (subscriber, #359)
[Link]
Posted Nov 26, 2018 21:46 UTC (Mon)
by karkhaz (subscriber, #99844)
[Link] (1 responses)
Also, of course the into this: I do wish the keyword was
Posted Nov 27, 2018 6:47 UTC (Tue)
by marcH (subscriber, #57642)
[Link]
https://www.google.com/search?q=early+return+perfect+pint...
Posted Nov 26, 2018 21:51 UTC (Mon)
by excors (subscriber, #95769)
[Link] (2 responses)
As long as the complex loops are still essentially iterating, I think that's the clearest way to write them. 'while' loops don't express the same intention - you have to read them more carefully to understand what they're trying to do.
But it's at least partly a matter of taste, so people will disagree. But most people will probably agree that consistency is valuable, and in the absence of an objective optimal style, it seems reasonable for maintainers to apply their own consistent taste to patches.
Posted Nov 26, 2018 22:25 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (1 responses)
Posted Nov 27, 2018 7:07 UTC (Tue)
by marcH (subscriber, #57642)
[Link]
The comment you answered tried to explain what a number of developers[*] try to communicate to each other _in the cases when_ they choose a for loop over some other equivalent style. Those numbers are unrelated.
[*] apparently not including you; not a problem
> hence, there is no "consistency" in this respect.
Not sure what "consistency" you're referring to but it wasn't the one excors mentioned.
Posted Nov 26, 2018 22:58 UTC (Mon)
by nix (subscriber, #2304)
[Link]
Given that 24-hour beginners' introductions to C describe the third arg to for as being for incrementing loop counters, I don't know why you think you're telling the readers here anything by revealing that as if it were some piece of obscure writ.
Posted Nov 27, 2018 7:44 UTC (Tue)
by marcH (subscriber, #57642)
[Link]
> The changes were mostly cosmetic
Not nice but not the end of the world either...
> The net effect was that my name ended up being attached to some code I didn't write ...
For the record: in my previous answer I assumed the maintainer stole the credit from you.
Posted Nov 21, 2018 6:16 UTC (Wed)
by k8to (guest, #15413)
[Link]
I guess it's possible to do better, but it's not common.
Posted Nov 20, 2018 6:24 UTC (Tue)
by alison (subscriber, #63752)
[Link] (2 responses)
How long until kernel development doesn't need people at all? There will just be Linus and a slew of bots.
Posted Nov 21, 2018 2:40 UTC (Wed)
by jkingweb (subscriber, #113039)
[Link] (1 responses)
Posted Nov 21, 2018 18:23 UTC (Wed)
by dr@jones.dk (subscriber, #7907)
[Link]
Posted Nov 21, 2018 19:37 UTC (Wed)
by aparri (guest, #127998)
[Link]
Luckily enough people have been *testing* "code is documentation" on a daily basis; ;-) results vary, but certainly include "Wait, what? why?" and "Please don't mess with me!".
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
- some similar bug report or experience
- vaguely/unconsciously remembering your idea but forgetting about you and your patch more specifically
These "higher authorities" must be pretty rotten, then, and this decidedly wasn't the case: The changes were mostly cosmetic, replacing a switch with an if - else - cascade and a while (or do-while) loop with a for(;;;) loop (IIRC), translating C into Go, so to say, considering that Go is basically Pidgin-C by design and intent. The net effect was that my name ended up being attached to some code I didn't write and wouldn't ever have written in this way for reasons I did (and do still) consider very good.
The kernel developer panel at LPC
IOW, this was mostly a case of BTNHIWHDI ("But that's not how I would have done it!"), the reason for this being "you didn't".
It's tempting to try an interpretation in the (loosely) given context here but I won't. This is something to think about (or not, at anybody's discretion).
The kernel developer panel at LPC
for (;;;) is the C attempt at imitating a proper counting loop and should never have seen the light of the day: It's basically a compiler macro which can be used to put some random parts of a loop body into its header (or all of it if so desired). Neither the first nor the last header part has any semantic constraints attached to it it's just "code which should run before the loop starts can be put here" and "code which should run after the loop body can be put there".
The first part, I'll grant you. The last part is very much not the case: it cannot be imitated by placing code either at the top of the loop body or after the loop (you have to duplicate it in both places).
Ritchie's original C reference manual defines a for loop as
The kernel developer panel at LPC
The for statement has the form
for ( expression-1opt ; expression-2opt ; expression-3opt ) statement
This statement is equivalent to
expression-1;
while ( expression-2 ) {
statement
expression-3 ;
}
and there's most certainly no semantic function associated with the third term. It's just an expression statement at the end of the loop body.
C also has a continue statement which can be used to skip all of the remaining loop body of a for-loop except expression-3. This means if continue is being used, expression-3 may be a convenient place for some sort of "step loop variables" action. OTOH, continue is rarley used (less so by people who routinely for-loop everything) and it may as well not. I've been writing C professionally for 15 years now and have yet to encounter a situation where for (;;;) offered something not as easily available without it. And the idea to put "statement that's to be executed after any other in the loop body" in front of all these statements isn't exactly great.
The kernel developer panel at LPC
{
int x = 1;
while(x)
{
--x;
}
}x go out of scope, to get the equivalent of this:for(int i = 1; i; --i) /* empty */ ;
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
(void)png_get_PLTE
(
/*png_ptr =*/ pngcontext,
/*info_ptr =*/ pnginfo,
/*palette =*/ &pngcolors,
/*num_palette =*/ &nrcolors
);png_get_PLTE(~png_ptr:pngcontext, ~palette:pngcolors, ~num_palette:nrcolors, ~info_ptr:pnginfo)
~info_ptr:info_ptr can be shortened to just ~info_ptr), to encourage programmers to give things the same name throughout the codebase if possible. So I do have sympathy with ingenious schemes to make code more friendly...but that thread was grotesque nightmare-fodder.
The kernel developer panel at LPC
png_get_PLTE(png_ptr=pngcontext, palette=pngcolors, num_palette=nrcolors, info_ptr=pnginfo)
for item in items:
if item == wanted:
break
else:
raise ItemNotFound
# now do stuff with item
You can do named parameters AND have the compiler check them (unlike comments), taking advantage of structs and the fact they are copied by value (and anon structs make this even easier to use than before):
The kernel developer panel at LPC
struct box_spec {
int height;
int width;
int depth;
};
box *create_box (struct box_spec spec) {
...
}
...
create_box ((struct box_spec) { .width = 10, .depth = 5, .height = 20 });
Note the order of the "arguments" no longer matters (unlike a function taking 3 ints).
The kernel developer panel at LPC
The kernel developer panel at LPC
continue statement is widely used in for-loops. When searching for something in a list that matches a series of criteria, you can use it to avoid multiple-nested conditional hell. Rewrite this:for(const auto &expr : exprs)
{
if(expr.is_boolean())
{
bool_exp = expr.to_bool_exp();
if(bool_exp.is_conjunction());
{
exp lhs = bool_exp.lhs();
exp rhs = bool_exp.rhs();
if(lhs.is_constant() && rhs.is_constant())
{
...
}
}
}
}for(const auto &expr : exprs)
{
if(!expr.is_boolean())
continue;
bool_exp = expr.to_bool_exp();
if(!bool_exp.is_conjunction());
continue;
exp lhs = bool_exp.lhs();
exp rhs = bool_exp.rhs();
if(!lhs.is_constant() || !rhs.is_constant())
continue;
...
}next instead of continue like it is in some other languages. Continue implies to me that you're going to continue executing the body of the loop, when what you're actually doing is bailing out early and going onto the next iteration.The kernel developer panel at LPC
https://lwn.net/Articles/723489/
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
C also has a continue statement which can be used to skip all of the remaining loop body of a for-loop except expression-3. This means if continue is being used, expression-3 may be a convenient place for some sort of "step loop variables" action. OTOH, continue is rarley used
continue certainly is not rarely used. A quick grep of the kernel suggests that continue is used perhaps 28% as often as for. Nearly 30,000 uses across the kernel tree is not rare in any sense. (Other codebases use it even more heavily). continue is useful for exactly the same reason return is: early exit if what you want to do is exit-to-the-next-iteration.
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC
The kernel developer panel at LPC