| |||||||||||||
unikernels and unified projectsunikernels and unified projectsPosted Mar 23, 2010 20:24 UTC (Tue) by mingo (subscriber, #31122)In reply to: Perhaps hubris is not a good programmer characteristic. by ejr Parent article: KVM, QEMU, and kernel project management
Similar things were said about Linux 15+ years ago, that its unikernel (non-microkernel) design that tied in all the kernel stack into a single unit was a horrible idea. That design worked out very well in practice, and we are seeing similar things with perf as well.
Qemu was dying before Linux/KVM reinvigorated it - this is an uncontested fact that was acknowledged by Avi too. Furthermore, Avi/Anthony stated it in the discussion that the location and splitup of the repository is immaterial. So if they are right, then the position of the repo within the Linux kernel repo is immaterial as well - and there would be little effect from such a move. If i'm right then it would reinvigorate Qemu and KVM even more. It's all speculation though - we wont know it for sure until we've tried it. (Which is moot and wont happen as per the current position of the KVM and Qemu maintainers.) Thanks, Ingo.
(Log in to post comments)
unikernels and unified projects Posted Mar 23, 2010 20:58 UTC (Tue) by ejr (subscriber, #51652) [Link]
Yes, in another 5-10 years, perf may have all the features of other, existing tools. Good example.
unikernels and unified projects Posted Mar 23, 2010 21:38 UTC (Tue) by mingo (subscriber, #31122) [Link] Nah, no need to wait 5-10 years, as perf already exceeds Oprofile in quite a few areas: - PEBS (preicise profiling) hw support on Intel CPUs - supports multiple, overlapping profiling sessions done at once - supports precise per task and per workload profiling (Oprofile only does system-wide profiling) - working call-graph profiling - integration with tracepoints (for example you can profile based not on some hardware counter such as cycles, but based in "IO submitted" events) - 'perf diff': view differential profiles - 'perf lock': lock profiling - 'perf sched': scheduler profiling/tracing - 'perf kmem': SLAB subsystem profiling - 'perf probe': dynamic probe based tracing, profiling - 'perf timechart': analyse delays - superior usability ;-) So yes, while it's still a very young project, it's working quite well and is growing fast. Also, i'm curious, which particular features/capabilities are you looking for? Thanks, Ingo
unikernels and unified projects Posted Mar 23, 2010 22:17 UTC (Tue) by ejr (subscriber, #51652) [Link]
Again, OProfile is not the right target. It has never been a player outside Linux kernel folks. You refuse to do your homework and keep pulling out strawmen. Look up Paradyn, TAU, and PAPI.
unikernels and unified projects Posted Mar 23, 2010 22:49 UTC (Tue) by mingo (subscriber, #31122) [Link] Again, OProfile is not the right target. It has never been a player outside Linux kernel folks. [...] Oprofile was used well beyond the kernel: it was the main profiler used for glibc development, Xorg development, Apache and many other OSS projects. It was what drove Linux PMU development, so naturally we concentrated on those usecases.If you are arguing that those are not important then i disagree with you. (/dev/perfctr on the other hand was an external driver that never saw enough use to find someone to push it upstream.) [...] You refuse to do your homework and keep pulling out strawmen. Look up Paradyn, TAU, and PAPI. FYI, PAPI has already been ported to the perf syscall, and both TAU and Paradyn are PAPI based - so they should work. I cannot see wide use of them beyond the HPC world, can you cite examples of big OSS projects making use of them? In any case tools/perf/ does not try to be the exclusive user of the perf syscall - there are other user-space tools that make use of it. Also, could you please skip the insults and the condescending tone? Thanks, Ingo
unikernels and unified projects Posted Mar 24, 2010 18:29 UTC (Wed) by deater (subscriber, #11746) [Link]
I'm still waiting for perf to have half the features of the pfmon tool that came with perfmon2.
unikernels and unified projects Posted Mar 26, 2010 9:43 UTC (Fri) by mingo (subscriber, #31122) [Link]
I know pfmon and as far as i'm aware perf currently has a lot more features than pfmon ever had. That's obviously in stark contrast with your statement - so could you please cite the features that perf doesnt have? Either we missed them (and thus will implement them) or they are implemented but you are not aware of them. Thanks, Ingo.
unikernels and unified projects Posted Mar 26, 2010 21:41 UTC (Fri) by deater (subscriber, #11746) [Link]
ability to specify _all_ events by name, i.e. if I want "SNOOPQ_REQUESTS_OUTSTANDING.DATA" I can just say so, instead of having to look it up and then having to specify the unintuitive "r01b3".
Being able to specify an output file that is nice ascii text, dumped every time a counter hits a certain threshold. This is probably possible with perf, but the perf --help is hard to follow.
It would be nice to have a perf-only mailing list, but I guess that will never happen.
It would also be nice to be able to build perf without having to download the entire kernel tree, I often don't have the space or the bandwidth for hundreds of megabytes of kernel source when I just want to quick build perf on a new machine.
unikernels and unified projects Posted Mar 23, 2010 22:22 UTC (Tue) by ejr (subscriber, #51652) [Link]
Oh, and more specifically: Dynamically instrumenting user-space code with performance counters. That's existed since '94 or so and has been deployed on multiple OSes and large-scale systems with higher level tools and other gizmos.
unikernels and unified projects Posted Mar 23, 2010 23:02 UTC (Tue) by mingo (subscriber, #31122) [Link] Most of those tools are PAPI based, and PAPI has been ported to the perf syscall as well - so there should be no problem whatsoever. So, tools/perf/ is in no way an exclusive user of performance events - if that is your worry/argument. It uses the perf events system call, and that system call has a strict ABI and other tools can make use of it (and are making use of it today). Thanks, Ingo
Finding a profiler that works, damnit Posted Mar 23, 2010 23:19 UTC (Tue) by foom (subscriber, #14868) [Link] > - working call-graph profilingReally? Over the last week, I've tried just about every profiler I can get my hands on, in hopes of finding ONE that works properly on x86-64. And I did find one. Just one. And it doesn't do system profiling. If you know some secret way of making perf actually work, do tell. gprof: Old standby -- I used to use this all the time back in the day... Unfortunately, only works on staticly linked programs. Not only does it not report functions in shared libraries, it simply IGNORES any samples where the PC was outside the staticly linked area entirely. So, pretty much useless. sprof: Doesn't even work at all. Not even a little bit. Completely useless, waste of disk space. oprofile: Works fine unless you want callgraphs. Then you need to recompile everything with - fno-omit-frame-pointers. Then it works okay. Also, can get *really* confused when you have chroots, which is rather annoying. Not particularly practical, due to recompilation requirement. Also found no way to convert to callgrind format (except a hacky script which only does flat profiles) perf: I upgraded to kernel 2.6.32, and installed this, cause I heard it was the great new thing.... but it *also* only seems to output proper callgraphs when programs (and all libraries...) are compiled with -fno-omit-frame-pointer. Furthermore, even when having done that, I can't make heads or tails of the callgraph output report. It is almost 100% unintelligible. It looks nothing like any callgraph profile I've ever seen before. I also found no way to convert to output to callgrind format. Bonus fail: ungoogleable name. valgrind callgrind: Best so far. Has a *GREAT* UI (kcachegrind), and outputs a reliable profile. Downside: program runs 100 times slower. (ARGH!). google-perftools libprofile: Seems to actually work. Hooray! Bonus: Outputs callgrind format, so I can use kcachegrind. Only downside: no system profiling.
Finding a profiler that works, damnit Posted Mar 24, 2010 0:30 UTC (Wed) by chantecode (subscriber, #54535) [Link] but it *also* only seems to output proper callgraphs when programs (and all libraries...) are compiled with -fno-omit-frame-pointer.
Sure, how could it be another way? Without frame pointers you can't have reliable stacktraces. Or if you have a tip to go round this requirement I would be happy to implement it. The only arch I know that is able to walk the stack correctly without frame pointers is PowerPc. Furthermore, even when having done that, I can't make heads or tails of the callgraph output report. It is almost 100% unintelligible. It looks nothing like any callgraph profile I've ever seen before. The default output is a fractal statistical profile of the stacktraces, starting from the inner most caller (the origin of the event) to the outer most. Let's look at a sample: http://tglx.de/~fweisbec/callchain_sample.txt What this profile tells you is that the ls process, while entering the kernel, is part of 3.34 % of the total overhead, and the origin of this overhead is in the __lock_acquire function. Among all the callers of __lock_acquire() when it caused this overhead, lock_acquire() has been its caller 97 % of the time, and _raw_spin_lock() has been the caller of lock_acquire() 48 % of the time, etc... This is why it is called a fractal profiling: each branch is a new profile on its own. _raw_spin_lock() is profiled relatively to its parent lock_acquire(). May be it doesn't look like other kind of callgraph profiler as you said. I just don't know as I haven't looked much at what other projects do. May be I took some inspiration from sysprof callgraphs, except sysprof does a outer most to inner most callgraph direction. The other direction we took for perf (from inner to outer) seems to me much more natural as we start from the hottest, deepest, most relevant origin to end up on the highest level origin. But I can implement the other direction, shouldn't be that hard, I'm just not sure it will give us nice and useful results but it's worth the try, I think I'll put my hands on it soon. BTW, we have a newt based TUI that can output the callgraph in a collapsed/expanded (toggable at will) fashion. May be that could better fit your needs. An example here: http://tglx.de/~acme/perf-newt-callgraph.png But you need to fetch the -tip tree for that as it's scheduled for 2.6.35 (git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-tip.git) But please if you have suggestions to make our callgraphs better, tell us. It would be very appreciated, we lack feedbacks in this area and it's still a young feature (although pretty functional). I also found no way to convert to output to callgrind format. You're right. I'll try to get this too once I get more time.
Finding a profiler that works, damnit Posted Mar 24, 2010 1:11 UTC (Wed) by nix (subscriber, #2304) [Link] Sure, how could it be another way? Without frame pointers you can't have reliable stacktraces.GDB has been able to use DWARF2 information to produce stack traces without frame pointers for many releases now.
Finding a profiler that works, damnit Posted Mar 24, 2010 1:25 UTC (Wed) by HelloWorld (guest, #56129) [Link]
How does that help? The problem seems to be that one has to recompile a lot of stuff in order to get stack traces. You have to recompile in order to get dwarf2 debugging information too.
Finding a profiler that works, damnit Posted Mar 24, 2010 3:12 UTC (Wed) by joseph.h.garvin (subscriber, #64486) [Link]
It's slightly more usable, because people are usually doing debug builds already, but not -fno-omit-frame-pointer builds. It's also not uncommon to release optimized builds with debug symbols to make cores easier to understand in the wild.
Finding a profiler that works, damnit Posted Mar 24, 2010 8:12 UTC (Wed) by ringerc (subscriber, #3071) [Link]
... and often those debug symbols are included as separate -dbg or -debug packages in the package management system, so if your tool knows how to read gdb external debug symbol files, you get to use the original binaries AND get debug info.
Of course, many profiling tools don't know about external debuginfo files...
Finding a profiler that works, damnit Posted Mar 24, 2010 10:31 UTC (Wed) by nix (subscriber, #2304) [Link]
Virtually everything that reads debugging information knows how to use external debug files, both in /usr/lib/debug/$PATH.dbg format and in /usr/lib/debug/$BUILD_ID format, because both of these have been used by widely-released distributions (Ubuntu and Red Hat have both used various versions of these).
It's been some years since I've come across any tool that reads DWARF2 that doesn't know about at least one of these, and in the last year or so everything has learnt about both (including perf).
Since pretty much every distribution is now compiling everything with separated debugging information, anything that doesn't support it isn't going to be particularly useful.
Outside rare code (e.g. the occasional garbage collector) that actually needs frame pointers for correct operation, -fno-omit-frame-pointer is needless, and using -fomit-frame-pointer instead gives a substantial speedup on many common platforms. IIRC it's the default on x86-64, but I don't have a recent GCC source tree here so I can't be sure...
Finding a profiler that works, damnit Posted Mar 25, 2010 4:23 UTC (Thu) by adsharma (subscriber, #8967) [Link]
libunwind supports multiple architectures and can unwind without frame
Finding a profiler that works, damnit Posted Mar 24, 2010 1:55 UTC (Wed) by foom (subscriber, #14868) [Link] >Sure, how could it be another way? Without frame pointers you can't have reliable stacktraces. Or if you have a tip to go round this requirement I would be happy to implement it. The only arch I know that is able to walk the stack correctly without frame pointers is PowerPc.GCC always includes an eh_frame section (unless you explicitly pass - fnoasynchronous-unwind-tables, which of course you should never do), which allows you to unwind reliably from any point within a program. This is actually required by the X86-64 ABI. This data is never stripped, and actually is mapped into memory with the code and data of the executable, so you can read it directly from the process's memory space. It's almost the same format data as the DWARF debug info, but with a couple minor differences and with only a subset of the functionality allowed. See libunwind for a userspace library which knows how to read and interpret this data. Google-perftools uses libunwind, which is how it's able to work properly.
Finding a profiler that works, damnit Posted Mar 24, 2010 2:25 UTC (Wed) by chantecode (subscriber, #54535) [Link]
Ok.Thanks for this pointer, I had no idea about this.
It doesn't look usable by perf though.
We often save the stacktraces from an NMI path (covers all hardware pmu events). So this operation must be as fast and simple as possible.
This seems to me really too complicated, too slow and unlikely to work most of the time.
Finding a profiler that works, damnit Posted Mar 24, 2010 3:19 UTC (Wed) by foom (subscriber, #14868) [Link]
> This seems to me really too complicated, too slow and unlikely to work most of the time.
Shrug. Sure, it's complicated, but it's totally necessary for the callgraph profiler to be useful at all
If you don't think it's possible to implement support for EH_FRAME-based unwind, you might as
Finding a profiler that works, damnit Posted Mar 24, 2010 10:34 UTC (Wed) by nix (subscriber, #2304) [Link]
In any case, the fix is obvious. Dump an address out in the fast path, and convert it to a stacktrace later on. This is what absolutely everything else does, from GDB to valgrind to sysprof.
Finding a profiler that works, damnit Posted Mar 24, 2010 15:18 UTC (Wed) by foom (subscriber, #14868) [Link]
> Dump an address out in the fast path, and convert it to a stacktrace later on.
...huh?
You can't convert a single address to a stacktrace later on! You'd need a copy of the whole stack to
Anyways, thanks for mentioning sysprof: I hadn't heard of that one before. But looking at the
Finding a profiler that works, damnit Posted Mar 24, 2010 17:48 UTC (Wed) by nix (subscriber, #2304) [Link]
Oh crap, you're right, of course. I plead temporary insanity caused by hours of sitting through mind-numbing presentations on stuff I already knew.
So it's call the unwinder or nothing, really. Unfortunately the job of figuring out what the call stack looks like really *is* quite expensive :/ any effort should presumably go into optimizing libunwind...
Finding a profiler that works, damnit Posted Mar 26, 2010 17:55 UTC (Fri) by sandmann (subscriber, #473) [Link] I am the author of sysprof.You are right that it doesn't generate good callgraphs on x86-64 unless you compile the application with -fno-omit-frame-pointer. I very much would like to fix this somehow, but I just don't see any really good way to do it. Fundamentally, it needs to take a stack trace in the kernel in NMI context. You cannot read the .eh_frame information at that time because that would cause page faults and you are not allowed to sleep in NMI context. Even if there were a way around that problem, you would still have to *interpret* the information which is a pretty hairy algorithm to put in the kernel (though Andi Kleen did exactly that I believe, resulting in flame wars later on). You could try copying the entire user stack, but there is considerable overhead associated with that because user stacks can be very large (emacs for example allocates a 100K buffer on the stack). You could also try a heuristic stack walk (which is what an old version of sysprof - new versions use the same kernel interface as perf). That sort of works, but there is a lot of false positives from function pointers and left-over return addresses. The function pointers can be filtered out, but the return addresses can't. These false positives tend to make sysprof's UI display somewhat confusing, though not completely unusable. You could also try some hybrid scheme where the kernel does a heuristic stack walk and userspace then uses the .eh_frame information to filter out the false positives. This is what I think is the most promising approach at this point. Some day I'll try it. Finally, the distributions could just compile with -fno-omit-frame-pointer by default. The x86-64 is not really register-starved so it wouldn't be a significant performance problem. The Solaris compiler does precisely this because they need to take stack traces for dtrace. But, I fully expect to be told that for performance reasons we can't have working profilers.
Finding a profiler that works, damnit Posted Mar 26, 2010 18:06 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]
Have you tried asking about that in fedora devel list? Maybe, we can change
the compiler options for Fedora 14.
Finding a profiler that works, damnit Posted Mar 26, 2010 21:44 UTC (Fri) by foom (subscriber, #14868) [Link]
I'm probably missing something, but...
Why does it need to happen at NMI time? Why can't you just do it in the user process's context,
The setitimer(ITIMER_PROF) solution that userspace profilers use clearly works out fine for
The stack trace of the userspace half clearly can't change between when you received the NMI and
That just leaves the complication of implementing the DWARF unwinder in the kernel, but there's
Finding a profiler that works, damnit Posted Mar 26, 2010 23:28 UTC (Fri) by nix (subscriber, #2304) [Link]
There is already a DWARF unwinder in the kernel (or was, and it could be
resurrected). The tricky part is making it paranoid enough to be non-DoSable, even by hostile generators of DWARF2. IIRC the kernel unwinder was ripped out by Linus because it kept falling over when unwinding purely kernel stack frames...
Finding a profiler that works, damnit Posted Mar 27, 2010 14:10 UTC (Sat) by garloff (subscriber, #319) [Link]
Finding a profiler that works, damnit Posted Mar 27, 2010 15:12 UTC (Sat) by foom (subscriber, #14868) [Link]
Care to expand upon that link with some explanatory text?
Finding a profiler that works, damnit Posted Mar 28, 2010 0:26 UTC (Sun) by garloff (subscriber, #319) [Link]
Sorry, that was somewhat terse. The Novell Kernel Debugger has stack
unwinding features built-in; so this is something that might be leveraged in another project.
Finding a profiler that works, damnit Posted Apr 4, 2010 12:35 UTC (Sun) by chantecode (subscriber, #54535) [Link]
We need to profile from NMI if we want to profile irqs as well. Otherwise a hardware pmu event would occur at the end of an irq disabled section, not at the exact place of the event, messing completely the result.
Finding a profiler that works, damnit Posted Apr 5, 2010 1:37 UTC (Mon) by foom (subscriber, #14868) [Link]
You need to get stack traces for the *kernel* from an NMI. Surely the userspace backtracing can
wait until a more convenient time...
Finding a profiler that works, damnit Posted Mar 30, 2010 18:51 UTC (Tue) by fuhchee (subscriber, #40059) [Link]
In systemtap, we do unwinding of kernel/userspace under similar constraints. We work around the "can't page in user data" by preemptively uploading the unwind data into the kernel, so it's ready for use when needed. It costs some memory but it saves time.
Finding a profiler that works, damnit Posted Mar 24, 2010 2:15 UTC (Wed) by foom (subscriber, #14868) [Link]
BTW, the callgraph output I'm used to reading is like this:
http://www.cs.utah.edu/dept/old/texinfo/as/gprof.html#SEC6
Or this:
The important points being that it shows, linearly, a section of output for every function, sorted by
Maybe your method is better for some uses, but I can't quite get my ahead around how to
Finding a profiler that works, damnit Posted Mar 24, 2010 2:48 UTC (Wed) by chantecode (subscriber, #54535) [Link]
Ah I see what you mean. We have this format too. We call that "flat" callchains. It's a sequence of traditional callgraphs, linear and sorted by overhead.
Example: http://tglx.de/~fweisbec/callchain_flat.txt
We have three kind of formats: flat (as in above link), fractal (as shown in my previous example) and graph (like fractal, except that percentages of overhead are absolute wrt to the whole overhead profiling, unlike fractal where each overhead percentage is relative to its parent branch).
syntax: perf report -g flat|fractal|graph
I've never heard about someone using the flat format. Everybody seem to use the fractal (probably because it's the default, and IMHO, the most useful). And because I've never heard about it, I've not tested it for a while so it works on 2.6.32 but seem to have been broken in 2.6.33 (the graph format too), but fractal works. I'm going to send a fix for that and get it to the 2.6.33 stable branch.
>Maybe your method is better for some uses, but I can't quite get my ahead around how to use it, even after your explanation of what it means. I don't expect I'm unique in this regard, since basically all callgraph profilers I've seen work the same way.
I must confess it's the first time I hear this complaint. That said may be it's a matter of playing a bit with it before being used to its layout.
Finding a profiler that works, damnit Posted Mar 24, 2010 3:41 UTC (Wed) by foom (subscriber, #14868) [Link]
> I've never heard about someone using the flat format.
I'm not surprised about that: it looks like it's basically just the same as fractal, except without
I just like to see a list of the *direct* callers and callees of every function. (having both easily
I think the gprof docs explain its format pretty well, and that's basically exactly what I want to
Or, heck, if you get callgrind output conversion working, kcachegrind also presents the data in
Finding a profiler that works, damnit Posted Mar 25, 2010 2:52 UTC (Thu) by chantecode (subscriber, #54535) [Link]
Ah ok, I see what you mean know. I parsed too quickly the gprof documentation.
Yep, may be we can implement such layout mode too, or at least getting it through kcachegrind.
Finding a profiler that works, damnit Posted Apr 1, 2010 20:42 UTC (Thu) by oak (subscriber, #2786) [Link]
Having support for perf callgraph format in this:
http://code.google.com/p/jrfonseca/wiki/Gprof2Dot
Or similar callgraph visualization support in perf itself would be nice (including filtering options, highlighting the nodes based on CPU used by given node and the nodes called by it etc).
Finding a profiler that works, damnit Posted Apr 4, 2010 12:44 UTC (Sun) by chantecode (subscriber, #54535) [Link]
I really like this. We may indeed want to support it.
Finding a profiler that works, damnit Posted Apr 5, 2010 1:39 UTC (Mon) by foom (subscriber, #14868) [Link]
Check out the google-perftools' pprof's "gv" command; it's also pretty nice. (although I still usually
prefer kcachegrind's interactive viewer)
Finding a profiler that works, damnit Posted Mar 24, 2010 7:11 UTC (Wed) by njs (guest, #40338) [Link]
> Also found no way to convert to callgrind format (except a hacky script which only does flat profiles)
Here's a hacky script that does callgraphs: http://roberts.vorpus.org/~njs/op2calltree.py
Finding a profiler that works, damnit Posted Mar 24, 2010 8:10 UTC (Wed) by ringerc (subscriber, #3071) [Link] gprof: Old standby -- I used to use this all the time back in the day... Unfortunately, only works on staticly linked programs. gprof works fine on dynamically linked programs. In fact, it's awfully rare to find a genuinely statically linked program these days - most "static" programs dynamically link to libc via ld-linux.so and statically link in all the other libraries they use, as ldd will reveal. Not always, but usually. I suspect what you were going for was "gprof only profiles one particular library or executable at a time, not a whole program including any shared libraries or plugins it uses." (Even that's not strictly true, it's just that you have to rebuild those plugins or shared libraries with gprof instrumentation too). I'm totally with you on the general "argh" re callgraph profiling across multiple programs or a subset of the system, though.
Finding a profiler that works, damnit Posted Mar 24, 2010 15:05 UTC (Wed) by foom (subscriber, #14868) [Link]
> Even that's not strictly true, it's just that you have to rebuild those plugins or shared libraries
> with gprof instrumentation too.
No, gprof really just doesn't work at all with shared libraries. If you link an executable with -pg,
So if you do run gprof with dynamically linked executable, your timing measures will all be
|
|||||||||||||