OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 7:57 UTC (Wed) by mjthayer (guest, #39183)
In reply to: OpenSolaris governing board threatens dissolution (The H) by drag
Parent article: OpenSolaris governing board threatens dissolution (The H)

> I know that people have worked on tracing features for Linux, but they are not really up to the same level that Dtrace is.
How does SystemTap compare these days? I tried both out for the first time recently, and after the slight difference in setup difficulty (no setup for DTrace vs two hours for SystemTap on Fedora 13 and half a day, despite the experience from Fedora, on Ubuntu 10.04) I didn't see any immediate difference. I assume that the differences come out when one moves to more advanced use. And of course SystemTap does feel horribly clumsy with the way it builds kernel modules out of trace scripts, although in some ways it is a rather nice idea.

> Besides that it's all personal preferences and prejudices that some people have against Linux/for Solaris.
I have been told that Solaris's tools for heavy-duty system administration knock Linux's into a cocked hat, but since I am pretty clueless about system administration I can't really judge that. Perhaps I should be buying and reading the ULSAH, as per yesterdays LWN article, to broaden my horizons a bit...

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 10:24 UTC (Wed) by mjthayer (guest, #39183) [Link] (1 responses)

>> I know that people have worked on tracing features for Linux, but they are not really up to the same level that Dtrace is.
>How does SystemTap compare these days?
Replying to myself, user space tracing with SystemTap is clearly not yet generally available, even if it is looking more hopeful again that it will be some day soon (c.f. https://lwn.net/Articles/387257/) . I have already made good use of user space tracing on OpenSolaris with DTrace.

Sweet SystemTap

Posted Jul 14, 2010 11:04 UTC (Wed) by mjw (subscriber, #16740) [Link]

Where it is available it is actually pretty nice. See for example this blog post on how to track reference counting bugs in gnome applications with SystemTap (and static trace points in the core gnome libraries):
http://tecnocode.co.uk/2010/07/13/reference-count-debuggi...
(The static trace points should even be source compatible with dtrace probes, so in theory this now also works for solaris.)

But yeah, it is taking some tricks to get all the support hooks accepted in the mainline linux kernel. Lots of pushing going on though. With tracehooks, ftrace, tracepoints and perf now being in we will hopefully see the more powerful systemtap features also trickle in over time. The realtime support also took a long time to get partially in even though lots of people were already using it outside mainline.

The next (1.3, not yet out) release of SystemTap can even give you user backtraces for when the kernel triggers a pagefault, which is pretty sweet. http://sourceware.org/git/?p=systemtap.git;a=blob;f=NEWS;...

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 12:13 UTC (Wed) by SEJeff (guest, #51588) [Link] (4 responses)

<snark>Because Jumpstart + the HORRIBLE package manager is soooooo much better than kickstart + rpm</snark> Even in OSOL, the package manager is terrible.

Can you please define "heavy duty systems administration". For server deployment en mass, Linux spanks the pants off of Solaris. Period.

For threading (especially if you have the Niagra sparc processor in your servers), Solaris still beats the pants off of Linux. Ditto with recovering from hardware (such as dimm) errors although that is more due to sparc being designed for those things and x86 not really until recently.

In the interest of full disclosureNote that SystemTap is all good and great, but isn't a full DTrace equivalent until the uprobes patchset is merged. Until it is, you can't do full on userspace <---> kernel space tracing, but only kernel space tracing.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 12:51 UTC (Wed) by mjthayer (guest, #39183) [Link] (3 responses)

> Can you please define "heavy duty systems administration". For server deployment en mass, Linux spanks the pants off of Solaris. Period.
Does that include updating the things and day-to-day maintenance?

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 13:19 UTC (Wed) by evad (subscriber, #60553) [Link] (2 responses)

Day-to-day maintenance could cover a multitude of things and I think this depends what you're doing with your OS - so comparing Linux and Solaris on that simple measure will always be difficult (and biased).

Updating Solaris vs. Linux systems is a lot easier. I'd argue that the easiest platform is Linux (yum, apt, etc), but Solaris isn't that much more difficult.

OpenSolaris and pkg make it very easy, and Solaris 10, although not quite as easy, is not at all difficult. Patch clusters and sets are easy to install and if you use pca its even easier. Unlike Linux systems you can use LiveUpgrade (although Fedora and btrfs are nearly there) and with patches you can roll patches back and know that its likely the rollback will work. On Linux with yum and apt you can, with difficulty, go back to older releases of packages but this isn't designed as so; you have no guarantee the process will work.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 13:34 UTC (Wed) by SEJeff (guest, #51588) [Link] (1 responses)

But you have the abomination of patches that have to be applied incrementally on Solaris vs a rpm you can upgrade directly to while skipping the intermediate versions. Thats why updating to the latest greatest from a base install can take eons. Please someone prove me wrong, my Solaris experience isn't near as extensive as my Linux experience.

It reminds me of the torvalds vs tanenbaum microkernel debate. They are great in theory, but horrible in practice and most implementations.

In the end, posix is posix is posix. Any competent admin with a posix skillset can manage both.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 14:50 UTC (Wed) by ESRI (guest, #52806) [Link]

IMO, patching on Solaris is an abomination. Especially coming from the Linux world where you can easily "yum upgrade" and reboot at your convenience.

Sure, LiveUgrade helps with some of this, but unless you're using a ZFS root, it's still a pain, and the nature of how Sun/Oracle bundles patches leads to people cherry picking them and you end up with systems all over the spectrum in terms of fully-patched, partially patched, etc.

PCA is great, but it's not really a centralized tool (without some custom infrastructure).

Suffice to say the reality is the Linux environments end up being nearly always fully patched, while the Solaris environments lag behind and/or only spot apply patches that address specific bugs or security holes.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 12:22 UTC (Wed) by lkundrak (subscriber, #43452) [Link] (7 responses)

...vs two hours for SystemTap on Fedora 13

Just curious -- what did you spend the two hours with? For me this always worked:

yum install systemtap
debuginfo-install kernel

SystemTap setup

Posted Jul 14, 2010 12:37 UTC (Wed) by mjw (subscriber, #16740) [Link]

The debuginfo-install kernel is technically only needed for probing the kernel (which admittedly is often where the juicy events come from). If you are only doing user space probing you only need the relevant user space debuginfo.

Also SystemTap comes with "stap-prep" which does the above for the kernel case plus sanity checks (modulo a bug on Fedora 13, which has been fixed upstream: https://bugzilla.redhat.com/show_bug.cgi?id=596083)

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 12:49 UTC (Wed) by mjthayer (guest, #39183) [Link] (5 responses)

> Just curious -- what did you spend the two hours with? For me this always worked:
Finding the documentation for SystemTap, working out the procedure, including what you need to install at all, then trying to work out why I couldn't seem to get a kernel version as new as the debug symbols I had just installed (or debug symbols as old as the kernel I had) and trying to find a matching kernel. In the end it was just my inexperience with yum I think, I normally use Ubuntu.

Ubuntu who unfortunately still don't have debug symbols matching their released kernel version (just the upcoming one), so that when I installed it on my Ubuntu system I ended up rebuilding the package in the background when I installed. Wouldn't have been a big issue if it hadn't already been so hot in the office... Admittedly not quite half a day, but several hours in total from start until I could actually use it.

SystemTap documentation

Posted Jul 14, 2010 12:55 UTC (Wed) by mjw (subscriber, #16740) [Link]

The Beginner Guide is pretty nice:
http://sourceware.org/systemtap/SystemTap_Beginners_Guide/

Other documentation on the sourceware homepage:
http://sourceware.org/systemtap/documentation.html

Debian is not also starting to ship debuginfo for the kernel, hopefully Ubuntu will pick that up:
http://www.researchut.com/blog/archive/2010/07/08/systemt...

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 15:26 UTC (Wed) by mjthayer (guest, #39183) [Link] (3 responses)

> Ubuntu who unfortunately still don't have debug symbols matching their released kernel version (just the upcoming one), so that when I installed it on my Ubuntu system I ended up rebuilding the package in the background when I installed. Wouldn't have been a big issue if it hadn't already been so hot in the office... Admittedly not quite half a day, but several hours in total from start until I could actually use it.

Oh yes, not to mention that due to some broken sanity check in SystemTap I always got

ERROR: Build-id mismatch: "kernel" vs. "vmlinux-2.6.32-23-generic"

when I tried to run a script. I took the lazy path and compiled the check out, as I didn't have time to understand the details of what it was doing, and it worked fine after that.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 15:09 UTC (Thu) by fuhchee (guest, #40059) [Link] (2 responses)

Please report the bug formally.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 15:29 UTC (Thu) by mjthayer (guest, #39183) [Link] (1 responses)

> Please report the bug formally.

Will do, I'm just always a bit embarassed reporting things I think I ought to be able to analyse myself without providing some sort of solution :)

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 10:27 UTC (Fri) by cesarb (subscriber, #6266) [Link]

There is nothing wrong with reporting something you ought to be able to analyze yourself but did not have the time to analyze. Besides, some things might take you hours or even days to fully analyze, but once pointed to a developer of the project, the developer sometimes is able to fully analyze and create a fix in minutes.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 14:53 UTC (Wed) by vonbrand (subscriber, #4458) [Link] (2 responses)

It has been quite a while since I futzed around with Solaris (around 2000). Solarites were known to belittle Linux for its lack of sysadmin tools back then. My experience was quite the contrary (with Red Hat in particular). Software updates are almost painless in Linux. Software availability was much better with Linux, either as packages or easy-to-build source (Solaris had its quirks, which required scary stuff like replacing libc for halfway sane operation, plus the customary "GNU > /usr/local" to get a reasonable shell and other basic tools, ...). The "easy to use" tools were such heavy resource hogs, and required Sun's bloated version of X, that they were completely useless in practice. Plus you couldn't even look sideways at the configuration files without giving them fits.

Besides, our Solaris server got compromised remotely due to a long-known vulnerability in said "administration tools", which we specifically had removed. A software update (re)installed them silently... Needless to say, an operating system than can be compromised remotely in a jiffy and stays that way for a couple of years isn't my first option. The machine was soon converted to Linux, and we never looked back.

Have had taking a new look at Solaris 10 on my ToDo list for some time... will set up a virtual machine to do so one of this days.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 15:53 UTC (Wed) by jwb (guest, #15467) [Link] (1 responses)

You know, my Solaris experience is WAY out of date, but when I was forming my opinion of it in 1999, it was just ghastly. Out of the box, the machines didn't even have FTP, nor sz/rz, so we had to use kermit to get an FTP client onto the machine and then FTP to transfer the rest of a sane userland onto the box. The compiler, which cost an abusrd amount of money, was inexplicably installed in /opt/SUNWSPro/blah/blah/blah/bin/, so everybody's PATH was a mess. It was basically a really bad experience.

Recently I decided to try Nexenta, the frankenmonster offspring of OpenSolaris and Debian, only to find that the userland there is a years-old fork of Ubuntu Hardy. This did not make me feel like the Solaris ecosystem had rocketed to greatness over the last decade.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 19:05 UTC (Wed) by mpr22 (subscriber, #60784) [Link]

But hey, at least it has nvi now.