This whole debate saddens me

Posted Nov 29, 2014 20:25 UTC (Sat) by Wol (subscriber, #4433)
In reply to: This whole debate saddens me by Richard_J_Neill
Parent article: The "Devuan" Debian fork

Except that NO OTHER INIT SYSTEM IS PORTABLE, EITHER !!!

Do any of the BSDs (other than Debian/BSD) use sysvinit?

Do any of the commercial nixen use sysvinit any more?

sysvinit only works on Debian/BSD thanks to a linux compatibility layer.

The systemd guys have taken the perfectly reasonable line that systemd is a linux init system, therefore relying on features unique to the linux kernel is a perfectly reasonable thing to do.

The trouble is, they see no problem in *relying* on *new* features, as other posters have said. That's likely to cause a backlash from the likes of RHEL/SLES, and once they start complaining loudly, changes are likely to happen ... :-) But again, their reasons for not wanting new features to be optional are good reasons - it adds bloat to have fallbacks ...

Cheers,
Wol

This whole debate saddens me

Posted Nov 29, 2014 22:24 UTC (Sat) by mgb (guest, #3226) [Link] (4 responses)

> Except that NO OTHER INIT SYSTEM IS PORTABLE, EITHER !!!

systemd is not just an init system. It aspired to replace the entire plumbing layer.

Making the entire plumbing layer non-portable would have been a huge step backward.

This whole debate saddens me

Posted Nov 29, 2014 22:46 UTC (Sat) by raven667 (subscriber, #5198) [Link] (3 responses)

That doesn't make any sense, the plumbing at this later is pretty much all kernel specific and non portable by definition.

This whole debate saddens me

Posted Nov 29, 2014 22:56 UTC (Sat) by mgb (guest, #3226) [Link] (2 responses)

> That doesn't make any sense, the plumbing at this later is pretty much all kernel specific and non portable by definition.

Which is why Debian GNU/kFreeBSD and Debian GNU/Hurd could not possibly exist.

But they do exist.

Systemd's non-portable plumbing layer would have been a huge backward step.

This whole debate saddens me

Posted Nov 30, 2014 0:27 UTC (Sun) by vonbrand (subscriber, #4458) [Link] (1 responses)

They exist because of BSD's Linux compatibility, because Hurd tries to be as like Linux as manageable (no own userspace). Besides, less than 1% of Debian counts as "doesn't exist" in my book.

This whole debate saddens me

Posted Nov 30, 2014 0:49 UTC (Sun) by rleigh (guest, #14622) [Link]

In end usage perhaps. But in actual useful development, you might be surprised at the proportion of development and infrastructure which was developed through the efforts of minor architectures and kernels, and the benefits which are brought by having them in terms of finding bugs which would be present but not noticed on the most commonly-used platforms. If they get dropped, it will lead to a loss of actively contributing developers and activities which do provide useful benefits to us all.

This whole debate saddens me

Posted Nov 29, 2014 22:58 UTC (Sat) by rleigh (guest, #14622) [Link] (3 responses)

Most other init systems are completely portable to other Unix systems because they use nothing but C and POSIX interfaces from libc. sysvinit is portable. It runs today in Debian on the Linux, FreeBSD and Hurd kernels with eglibc. It could undoubtedly run on other kernels and with other libcs such as ulibc.

The initscripts themselves do have some Linux-isms in them; primarily use of /proc which is catered for by linprocfs on FreeBSD and a translator (IIRC) on Hurd. For the scripts which are Linux-only (e.g. udev), that doesn't matter. For the others they could be cleaned up to be more portable if there was a need for that. E.g. I suspect use of /proc/mounts is no longer necessary now mtab has been eliminated; we could use the mount(8) output directly. If we wanted to remove the use of linprocfs and linsysfs, these scripts could be cleaned up just as we removed the bashisms. There hasn't been an realistic need for that, however.

I agree that the unconditional reliance on new features by systemd is a problem. You would have thought they would have the ability and resources to support conditional usage with a fallback, and to be able to test on older systems to make sure the fallbacks work.

This whole debate saddens me

Posted Nov 29, 2014 23:38 UTC (Sat) by raven667 (subscriber, #5198) [Link] (1 responses)

I think one important point though is that sysvinit is only portable to BSD and Hurd as far as those systems have linux compatibility layers, it isn't actually portable to the native APIs of those systems. Portability for applications is the same as it ever was, linux with systemd is just as different as OS X with launchd or Solaris with SMF or even linux with openrc, as much as an application needs to interact with the plumbing

This whole debate saddens me

Posted Nov 29, 2014 23:45 UTC (Sat) by misc (subscriber, #73730) [Link]

In fact, a good post on the topic :

https://teythoon.cryptobitch.de/posts/on-portability-of-i...

This whole debate saddens me

Posted Nov 29, 2014 23:44 UTC (Sat) by misc (subscriber, #73730) [Link]

Sometime, a fallback is not doable.

For example, the audit and containers issues that is discussed another place on this article's comment cannot have a "fall back". Smackd support cannot have a fallback. If kernel or something support is missing for a feature, then you cannot "fallback". Now, you can decide to not use it as a distribution and admins, but that's most of the time out of scope.

And stuff like cgroups did have a fallback ( from 12-04-2011 cfe5a53dc7 to may 2014, ie 3 years )

It is not like there is requirement just to annoy people..

This whole debate saddens me

Posted Nov 30, 2014 0:21 UTC (Sun) by vonbrand (subscriber, #4458) [Link] (21 responses)

Last time I looked, RHEL 7 was running systemd just fine, thank you. SUSE was also an early convert...

This whole debate saddens me

Posted Nov 30, 2014 0:59 UTC (Sun) by Wol (subscriber, #4433) [Link] (20 responses)

Yup. They're running systemd just fine. The problem is when the time comes to upgrade.

What's going to happen when some big company decides to upgrade their ancient RHEL7 system to RHEL9, hits kernel-related problems, and discovers they can't debug it because they can't change kernels?

You've only got to read the recent articles, and people are ALREADY complaining that this is a problem. Whether it gets worse or not depends on how the kernel and systemd camps decide to deal with it.

As has been pointed out, for systemd a two-year-old kernel is "too old to worry about". For RHEL/SLES, that kernel has several years life left. Which could make life very difficult if you need to swap kernels ...

Cheers,
Wol

This whole debate saddens me

Posted Nov 30, 2014 1:11 UTC (Sun) by vonbrand (subscriber, #4458) [Link]

Presumably RHEL 9 will still be using systemd...

This whole debate saddens me

Posted Nov 30, 2014 1:23 UTC (Sun) by mjg59 (subscriber, #23239) [Link] (14 responses)

No big RHEL customer is ever going to try to run an old kernel with a current RHEL userland. They've got support contracts exactly because they want to avoid them having to do that kind of crap themselves.

This whole debate saddens me

Posted Nov 30, 2014 17:18 UTC (Sun) by Wol (subscriber, #4433) [Link] (13 responses)

It's all very well saying "the customer has a support contract". The customer still has all the grief of a failed upgrade, and all you've done is MOVED the problem. You haven't cured it.

Even worse if it's an emergency upgrade (which if it's old hardware destined for scrap, is not unlikely).

Cheers,
Wol

This whole debate saddens me

Posted Nov 30, 2014 18:20 UTC (Sun) by vonbrand (subscriber, #4458) [Link]

Users with support contracts expect things to move smoothly from version N to N +1 ... it'part of the deal. And "emergency upgrade, machine is to be scrapped" hasn't ever come up in my experience...

This whole debate saddens me

Posted Nov 30, 2014 19:01 UTC (Sun) by mjg59 (subscriber, #23239) [Link] (11 responses)

No big company will respond to "RHEL 9 is broken" with "Let's run RHEL 9 on the RHEL 7 kernel". They'll respond by rolling back to the previously deployed RHEL and screaming at whoever was responsible for giving the go-ahead. In almost 5 years of dealing with RHEL kernel bugs, I literally never saw anybody attempting the situation you're describing. It just doesn't happen.

This whole debate saddens me

Posted Dec 2, 2014 18:49 UTC (Tue) by Wol (subscriber, #4433) [Link] (10 responses)

What I was thinking of, and we did that sort of thing on various occasions, was taking (as a real example from my past)

NT3.5 and UV9.6 on old hardware -> NT2000 and UV11 on new hardware.

This is exactly the thing Al Viro flew off on - if the new system performs badly relative to the old one, then you want to find out what the f*** went wrong!

And old hardware breaks. You might be planning an upgrade and your hand gets forced. Or manglement are skinflints and wait until something breaks. Or whatever.

It sounds like that has happened with PostgreSQL already, so it's not an "it might happen", it's a "We've already had to do this before".

Cheers,
Wol

This whole debate saddens me

Posted Dec 2, 2014 19:16 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (9 responses)

The equivalent here is "We ran Oracle 13 on RHEL 8 and it performs worse than Oracle 12 on RHEL 7, so we tried Oracle 13 on RHEL 7 and Oracle 12 on RHEL 8", not "We ran Oracle 12 on RHEL 8 and it performs worse than Oracle 12 on RHEL 7, so we tried running the RHEL 7 kernel under RHEL 8". You didn't try to diagnose Windows 2000 performance issues by running it on top of the NT 3.5 kernel!

This whole debate saddens me

Posted Dec 3, 2014 20:44 UTC (Wed) by Wol (subscriber, #4433) [Link] (8 responses)

But if they want to find the source of the problem, that's what you have to do. Change as few components as possible until you find what went wrong.

Given your example, if both Oracle 12 & 13 run fine on RHEL 7, and they run worse on RHEL 8, then you know the fault is RHEL. If running RHEL 8 systemd on RHEL 7, and RHEL 7 systemd on RHEL 8 also makes no difference, then you know it's the kernel. Then you play with the kernels until you find the one at fault.

But - given the tight coupling between systemd and the kernel - that sort of debugging is harder than it should be.

Linux' motto here is "don't break user space". If current systemd refuses to even boot when using *current* distribution kernels (even if they are decrepit kernels :-) that's a pretty serious user-space breakage!!!

Cheers,
Wol

This whole debate saddens me

Posted Dec 4, 2014 10:51 UTC (Thu) by mjg59 (subscriber, #23239) [Link] (7 responses)

Like I said, *nobody* does this. If performance is worse on RHEL 8, and you've verified that it's RHEL and not the application that's responsible, you complain to Red Hat.

This whole debate saddens me

Posted Dec 4, 2014 13:59 UTC (Thu) by madscientist (subscriber, #16861) [Link] (2 responses)

Er... *somebody* has to do it. If you file a bug with Red Hat do you think they just meditate on the code until the problem becomes obvious?

And let's remember that not everyone buys a support contract from a company, not everyone has extra systems lying around to test upgrades on before they upgrade their "important" systems, and not everyone does reinstall upgrades. Are we saying that these people are SOL and the fact that their life just potentially got a lot more complex so that systemd doesn't have to think about backward compatibility is ok, because those people "aren't doing the work" so who cares?

This whole debate saddens me

Posted Dec 4, 2014 19:52 UTC (Thu) by sjj (guest, #2020) [Link]

These days, *everybody* with a credit card has test systems available. Or model your stuff on your laptop in VMs. If your precious server is an Artisanal Build by your Veteran Unix Administrator, insist that its software configuration can be replicated from a version control system at any time.

This whole debate saddens me

Posted Dec 4, 2014 20:04 UTC (Thu) by mjg59 (subscriber, #23239) [Link]

The argument in http://lwn.net/Articles/623610/ was:

> What's going to happen when some big company decides to upgrade their ancient RHEL7 system to RHEL9, hits kernel-related problems, and discovers they can't debug it because they can't change kernels?

Everybody running RHEL has a support contract. Nobody running RHEL is going to debug it by swapping out individual components. Red Hat presumably feel comfortable in their ability to do that debug work.

This whole debate saddens me

Posted Dec 4, 2014 14:08 UTC (Thu) by dan_a (guest, #5325) [Link] (3 responses)

But then wouldn't Red Hat start doing all of that - because they will want to solve the regression?

This whole debate saddens me

Posted Dec 4, 2014 20:07 UTC (Thu) by mjg59 (subscriber, #23239) [Link] (2 responses)

My experience at Red Hat was that performance issues were generally diagnosed by looking at perf traces and making targeted attempts at fixing the issue rather than just bisecting - the parts of the kernel that tend to cause this issues are frequently rewritten between RHEL releases, so bisecting often just gives you "We changed this algorithm to this other algorithm", and you still need to figure out why it's pathological with the new implementation because just reverting that isn't an option.

This whole debate saddens me

Posted Dec 4, 2014 20:44 UTC (Thu) by viro (subscriber, #7872) [Link] (1 responses)

Do we, by any chance, have magical private perf patches that allow to obtain information about an apparent race leading to data corruption sometime reproducible by given test in xfstests, and whom should I ask for it? Just going by the last time I had to go into "bisect all the way to 3.0" mode...

</sarcasm> There's a whole lot of stuff other than performance regressions, as you damn well know. For those, yes, you want perf traces first and foremost (and even then you really might want to see where the hell has regression first happened - sometimes it helps in figuring out what's wrong with the current algorithm). And we do upstream work as well, as you also know - after all, crap happening in mainline eventually will end up as crap happening in the next RHEL branch. <sarcasm>

This whole debate saddens me

Posted Dec 4, 2014 21:02 UTC (Thu) by mjg59 (subscriber, #23239) [Link]

Undeniably. But they're still not cases where customers are going to try to swap out individual components themselves - the risk is customers being unhappy because it takes Red Hat longer to diagnose an issue, not customers being unhappy that they can't run hybrid RHEL systems.

This whole debate saddens me

Posted Dec 1, 2014 1:07 UTC (Mon) by foom (subscriber, #14868) [Link]

I'd be very, very, surprised if that even worked at all, systemd or not. Usually (at least historically) glibc gets compiled with a newer minimum kernel version, so, you'd have to rebuild glibc, too, and probably some other software too, if you wanted to actually run an old kernel.

This whole debate saddens me

Posted Dec 1, 2014 15:02 UTC (Mon) by filipjoelsson (guest, #2622) [Link] (2 responses)

Red Hat doesn't support upgrading major versions, do they? You migrate to new hardware, and pick the latest RHEL (or Centos), or you stay on the same hardware and it just keeps on ticking.

If you develop something new and shiny, and you want the latest features, it seems to me like a bad idea to put that on an old system which is already running critical tasks. Develop on Fedora, test and deploy on Centos/RHEL, after that don't touch anyhting apart from installing security updates and minor features.

If your product acquires brand new features worthy of a major version number change, then you already have a migration situation rather than an upgrade. Do it properly with new (virtual) hardware, test thoroughly, and deploy with the flip of a switch.

This whole debate saddens me

Posted Dec 1, 2014 15:36 UTC (Mon) by zdzichu (subscriber, #17118) [Link] (1 responses)

They do support upgrade from RHEL6 to RHEL7 (first systemd-shipping version) server edition: https://access.redhat.com/solutions/637583

This whole debate saddens me

Posted Dec 1, 2014 16:29 UTC (Mon) by filipjoelsson (guest, #2622) [Link]

Indeed, you are right and I was (partially) wrong. If you run the server edition on x86_64 hardware, there is an in-place upgrade path. If I read the linked page, the other editions and archs do not support it.

It seems to me that this would be the case, because in-place upgrade is a corner case, and its only worth supporting on the biggest platform's most expensive edition. Upgrading from RHEL5 to 7, which was the example I responded to, does still not seem to be supported.

Anyway, upgrading the OS on a production server to a new major version is still a Bad Idea[tm], for which reason the example of the OP was more than a little contrived. Or do you disagree?