Recent Features
| |||||||||||||
Poettering: The Biggest MythsPoettering: The Biggest MythsPosted Jan 28, 2013 21:27 UTC (Mon) by khim (subscriber, #9252)In reply to: Poettering: The Biggest Myths by davidstrauss Parent article: Poettering: The Biggest Myths
You can't possibly expect that the only way to challenge monolithic design is to run Hurd or some other microkernel. The only way to show that monolithic design is somehow bad is to show that alternative exist which is non-monolithic and better. There are exactly two classes of systems: Most popular OSes are all almost completely monolithic per this definition: Android, Busybox, BSDs. Only GNU/Linux is weird creature which includes robust and reliable monolithic part (GNU components: GCC, GLibC, binutils, coreutils, etc - they are not really supposed to be used with anything else) and loose and flaky things like sysvinit with it's plethora of weird scripts. Frankly: init is supposed to be small, simple and reliable part (it's tiny in comparison, to e.g. GLibC) but in reality it's least reliable part - and if you'll think this is exactly because it's not "monolithic". Systemd fixes that and it's a good thing. People have preached "write programs that do one thing and do it well" that they forgot the whole thing which says instead Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. How exactly can you convince programs to work together if you don't write them together? Unix was always created as a large cohesive set of tools which work together, not as a bazillion pieces to be assembled by the end user. Systemd does the same thing - yet somehow people try to say that's un-UNIX-like. How can the basic principle used by Unix for the last fourty years be un-UNIX-like?
(Log in to post comments)
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 21:47 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
> Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
This is exactly what is not happening. Instead systems are being written that try very hard to force you to replace many tools with this new system, and the components of the new system do not maintain compatibility with the old interfaces.
> How exactly can you convince programs to work together if you don't write them together?
this is a core part of the problem
you make programs work together, not by writing them together, but by defining the interface between them. That way the different programs can be written by different people at different times, replaced one by one, etc
That is the way to have a long-term robust solution.
If you consider that all the programs need to be written together, it's a very small step to consider that you can break all your old interfaces because you ship all new versions of everything.
Linux desktop environment projects are following this logic, and users quite rightly complain when major changes happen that break how they used to work, and make it so that you can't use old components that work great for you with the new version.
APIs need to be defined, and you need to support old ones going forward.
you do NOT need to write all your programs together
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 21:56 UTC (Mon) by raven667 (subscriber, #5198) [Link]
> This is exactly what is not happening. Instead systems are being written that try very hard to force you to replace many tools with this new system, and the components of the new system do not maintain compatibility with the old interfaces.
That's not something that's happening with systemd though as the developers have worked very hard on backwards compatibility including automatically importing /etc/init.d, parsing /etc/fstab, supporting initctl, supporting tools such as chkconfig. I'm not sure what more could possibly be done.
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 22:14 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
> I'm not sure what more could possibly be done.
don't merge udev, go ahead and enhance udev to provide whatever service you are wanting to use, but keep udev a separate project so that people who don't want systemd continue to be first-class users
don't reinvent syslog, if there is something that you need, contact the syslog maintainers and ask for the feature if you want it. Or if they don't have time to implement the feature, write it and contribute it (all of the syslog daemons accept outside contributions, and rsyslog, the default is very accepting, and supports sponsoring new features if you can't just convince them it's a good idea)
etc
for the most part it's not that systemd needed to do more, it's that they should have done less, writing utilities and working with existing tools rather than reinventing them or re-implementing part of the existing functionality inside systemd.
The latest thing with CRON is another example, the argument seems to be that they need to take over cron because it's another way to start a process, and they need to know about all process starts to 'manage' them.
How about instead defining a 'startup' program that starts jobs in their own cgroup, notifying interested programs through some mechanism (init being one of them, but probably not the only one) and then get CRON modified to use this program to fire off the apps.
This program could then be used by other things, desktop environments could use it to start apps, user scripts could use it to launch apps. Things besides init could watch for processes to start (such as high availability and other monitoring software)
This would be "the unix way", a tool that does one thing "launch an app with appropriate isolation and notification) that could be re-used by many other people
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 22:39 UTC (Mon) by davidstrauss (subscriber, #85867) [Link]
> don't merge udev, go ahead and enhance udev to provide whatever service you are wanting to use, but keep udev a separate project so that people who don't want systemd continue to be first-class users
It's sort of touched on in myth #10. Separate git repositories is not the only path to modularity.
> don't reinvent syslog, if there is something that you need, contact the syslog maintainers and ask for the feature if you want it. Or if they don't have time to implement the feature, write it and contribute it (all of the syslog daemons accept outside contributions, and rsyslog, the default is very accepting, and supports sponsoring new features if you can't just convince them it's a good idea)
Projects don't own turf just because they provide the current implementation. Contributing to existing projects is certainly a good method when possible, but I don't think you'd get very far asking MongoDB developers why they didn't try to get their changes progressively into PostgreSQL or MariaDB.
> The latest thing with CRON is another example, the argument seems to be that they need to take over cron because it's another way to start a process, and they need to know about all process starts to 'manage' them.
My same thoughts here. anacron didn't integrate their features into cron. Nor did chrony seek to slowly improve ntpd. Nor did git seek to build on Subversion. (And there are many more examples.) Why does systemd have a unique responsibility to improve other implementations rather than providing capable replacements?
> How about instead defining a 'startup' program that starts jobs in their own cgroup, notifying interested programs through some mechanism (init being one of them, but probably not the only one) and then get CRON modified to use this program to fire off the apps.
That sounds like a reasonable capability to me, but it doesn't have any bearing on whether systemd should implement other features like a cron equivalent and the journal.
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 23:17 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
> Separate git repositories is not the only path to modularity.
no, they are not. And separate git repositories do not guarantee modularity either (as an example, see the historic problems that have existed with ALSA user space vs kernel code)
There is a mindset required.
1. you have to accept that there will be other users of your interfaces
2. you have to accept that there will be people who don't use your favorite components
3. you have to define your interfaces
4. you have to maintain backwards compatibility for your interfaces (no dropping features just because it's convenient for you and you don't need them any longer)
> Projects don't own turf just because they provide the current implementation. Contributing to existing projects is certainly a good method when possible, but I don't think you'd get very far asking MongoDB developers why they didn't try to get their changes progressively into PostgreSQL or MariaDB.
did they even _try_ to work with existing projects? in the case of journal/syslog we know that they didn't, from the simple fact that a lot of the things that they claimed syslog couldn't do (and therefor justified the journal), were in fact easily done with syslog, and some of the other things were added within days of the journal announcement.
It's not like the journal added capabilities that are so shocking and new that they can't possibly work in standard syslog daemons.
> My same thoughts here. anacron didn't integrate their features into cron...Nor did git seek to build on Subversion....Why does systemd have a unique responsibility to improve other implementations rather than providing capable replacements?
git doesn't force you to not run subversion, being a system tool means that systemd needs to be more accomodating than user tools where multiple ones can be used.
in the case of anacron, you have a single purpose tool that replaced another single purpose tool, and did so in a way that supported existing configs. If systemd was just an init that fully supported everything that was in sysV init (it supports some, but not all, inittab for example) and was presented as "in this version we switch from init to systemd and you don't see any changes if you aren't using new features", and then added new ways of streamlining the things that init has been responsible for, you would not have nearly the pushback that you have now.
> That sounds like a reasonable capability to me, but it doesn't have any bearing on whether systemd should implement other features like a cron equivalent
in the case of CRON, if you were to take away the justification of needing to know about all the processes by having a 'startup' program, what's the justification for replacing a established, well known tool with a new implementation inside systemd?
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 23:51 UTC (Mon) by davidstrauss (subscriber, #85867) [Link]
> 1. you have to accept that there will be other users of your interfaces
Multiple external projects use the systemd interfaces. We've built a lot of them here at Pantheon. Some, like the Python journal API, have been moved to the systemd core repository. Some interfaces, like the PHP, Lua, and node.js interfaces, remain external.
Projects like the journal gateway are *entirely* APIs for external use and were built to improve inter-tool operation.
> 2. you have to accept that there will be people who don't use your favorite components
In the case of cron, there's no conflict at all. Keep running cron if you want to. It might eventually not be installed by default, but I'm sure it will be packaged and maintained for the next decade or so.
In the case of the journal, messages collected by systemd do have to travel through it but can be forwarded to syslog. syslog can also forward to the journal. The journal, by default, runs as an in-memory-only pass-through for messages to syslog.
In the case of init itself, you obviously can't have two co-existing as PID 1 on the same system, but systemd support init scripts, LSB headers, and the standard, traditional tools for managing services.
> 3. you have to define your interfaces
They're either plain text with key/value pairs (the journal interface), documented CLI tools, or metadata-defined D-Bus serializations.
> 4. you have to maintain backwards compatibility for your interfaces (no dropping features just because it's convenient for you and you don't need them any longer)
You can generally mix and match new and old API invocations without much trouble. For example, journalctl runs fine with newer and older journald implementations. The CLI tools have been consistent enough that we haven't had to alter Chef's systemd service support since it was implemented over a year ago. I'm not familiar enough with the D-Bus interfaces to comment on those.
> did they even _try_ to work with existing projects? in the case of journal/syslog we know that they didn't, from the simple fact that a lot of the things that they claimed syslog couldn't do (and therefor justified the journal), were in fact easily done with syslog, and some of the other things were added within days of the journal announcement.
First, you need to cite the things that were supposedly claimed impossible with syslog but were actually easy, including where systemd maintainers made the claim.
Second, a project doing something within days of a fork is not evidence they were receptive to the change before. The Linux kernel stonewalled Android-style sleep-as-default (I forget the exact name) until Android forked the kernel and demonstrated a successful architecture for it. It took more than a few days, but it's still clear that competitive pressure is quite different from lobbying a project.
> It's not like the journal added capabilities that are so shocking and new that they can't possibly work in standard syslog daemons.
Maybe, but nothing comparable (even in terms of major changes unrelated to the journal's goals) has happened with syslog daemons in the last decade of running Unix logging. Until the journal, I still couldn't log a PHP backtrace in any way that kept the lines all together.
> git doesn't force you to not run subversion, being a system tool means that systemd needs to be more accomodating than user tools where multiple ones can be used.
systemd timer units don't force you to not run cron. Stop spreading obvious FUD.
> "in this version we switch from init to systemd and you don't see any changes if you aren't using new features"
That was basically the case for Fedora aside from dropping rc.d symlinks. The journal only came later, and it runs as a completely separate daemon and CLI toolset.
> what's the justification for replacing a established, well known tool with a new implementation inside systemd?
Consistent configuration of executable units (whether it's a service or a periodic job) and consistent administrative tools for logging and monitoring success/failure. It's unclear to me why anyone would want to configure things the old cron way once they're accustomed to writing systemd units.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 2:54 UTC (Tue) by mathstuf (subscriber, #69389) [Link]
> There is a mindset required.
> 1. you have to accept that there will be other users of your interfaces > 2. you have to accept that there will be people who don't use your favorite components > 3. you have to define your interfaces > 4. you have to maintain backwards compatibility for your interfaces (no dropping features just because it's convenient for you and you don't need them any longer)
Well, systemd does that[1].
[1]http://www.freedesktop.org/wiki/Software/systemd/Interfac...
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 23:17 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
> How about instead defining a 'startup' program that starts jobs in their own cgroup, notifying interested programs through some mechanism (init being one of them, but probably not the only one) and then get CRON modified to use this program to fire off the apps.
Internally "cron" subsystem is implemented by a separate utility. It just uses the common systemd infrastructure and can be easily developed separately.
It's certainly possible to hook up systemd to launch services using DBUS interface. And you can then use it with any kind of triggers - I've actually used good old cron to start systemd tasks for a while.
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 23:22 UTC (Mon) by raven667 (subscriber, #5198) [Link]
> don't merge udev, go ahead and enhance udev to provide whatever service you are wanting to use, but keep udev a separate project so that people who don't want systemd continue to be first-class users
I'm not sure that this has anything to do with what we were discussion about systemd compatibility with existing management methods but I should point out that in this case udev is still very much supported as a stand-alone component although it is fair to mention that because it shares build scripts with the rest of systemd so there there are additional build time dependancies that aren't explicitly needed by udev alone. This is of interest to developers and Gentoo users but is of no practical effect to anyone else.
> don't reinvent syslog, if there is something that you need, contact the syslog maintainers and ask for the feature if you want it.
http://blog.gerhards.net/2011/11/journald-and-rsyslog.html
Some thoughts on the Journal by the main rsyslogd maintainer. What might be interesting is if a tool like rsyslog would talk the Journal protocol natively, possibly replacing the systemd included implementation for large environments. In the blog post there are several examples given as to why building a new component might be a better approach, because of the things that you can't change for compatibility reasons, even if you improve them in rsyslog.
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 22:20 UTC (Mon) by davidstrauss (subscriber, #85867) [Link]
> This is exactly what is not happening. Instead systems are being written that try very hard to force you to replace many tools with this new system, and the components of the new system do not maintain compatibility with the old interfaces.
First, it's not clear what loss of interface compatibility you're lamenting. systemd supports most of the same traditional CLI utilities, syslog protocol (both in and out of the journal), inetd-style service activation, and SysV-style init scripts (including the LSB metadata). The journal's CLI supports text streams compatible with all the standard shell tools.
That said, needing to replace many tools at once isn't inherently evidence of a monolithic design. It can also be a sign that the boundaries of functionality or interfaces between tools have changed.
For example, we added the journal with a new API and a compact on-disk format. Necessarily, we have to have a new daemon to receive messages (journald), a new tool for browsing the files as text streams (journalctl), and a new C header for doing the structured logging. The journal would be rather useless for a long time if added in program by program over years. Nothing in this design is more monolithic or less Unix-y than what it replaced, but it did require replacing all of those parts at once to get end-user value.
There is nothing in this statement about how many programs you should change at once:
systemd is not a single "program," nor is splitting an implementation over many version control repositories (a thing many people seem to fixate one) a sign of good, modular design.
> That way the different programs can be written by different people at different times, replaced one by one, etc
Many of the interfaces for systemd are well-defined: the D-Bus API, the native journal field format, discovery of inherited sockets for activation. Those latter two already have multiple implementations talking over the defined interfaces, often for scripting languages that don't want to link to the C functions systemd exports or for projects outside the main systemd repository. We're currently looking at having scripting languages like Python manage services directly over D-Bus as well to enforce stability and completeness for those interfaces.
The Python-based boot graph analysis tool that talks to systemd over D-Bus is about to get replaced with a completely new implementation in C by a different author.
So, systemd has already demonstrated the ability to substitute implementations interfacing over well-defined boundaries.
Therefore, I think it's unnecessary to replace programs one by one rather than as a suite. Nor does the Unix philosophy you quoted say anything about this concern.
you do NOT need to write all your programs together to make them work together. Posted Jan 28, 2013 23:23 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
> For example, we added the journal with a new API and a compact on-disk format. Necessarily, we have to have a new daemon to receive messages (journald), a new tool for browsing the files as text streams (journalctl), and a new C header for doing the structured logging. The journal would be rather useless for a long time if added in program by program over years. Nothing in this design is more monolithic or less Unix-y than what it replaced, but it did require replacing all of those parts at once to get end-user value.
and why did you need to change all of those things?
syslog daemons already supported outputting to many different on-disk formats, why should you be requiring a different format?
Why invent a new way of doing structured logging? there were already N standard ways of doing structured logging, why did you have to invent a new one for systemd?
etc....
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 0:00 UTC (Tue) by davidstrauss (subscriber, #85867) [Link]
> syslog daemons already supported outputting to many different on-disk formats, why should you be requiring a different format?
None of those approaches supports field-structured logging because syslog has no method for handling such messages being sent to the daemon, let alone preserving the data on disk after receipt. You could have syslog write to an RDBMS like SQLite for all it matters, but if the structure's already gone, the structure's already gone.
The journal format also has a number of nice attributes like FSS, compression, atomic rotation, and indexing. And it's all available as friendly text streams in many formats using journalctl.
> Why invent a new way of doing structured logging? there were already N standard ways of doing structured logging, why did you have to invent a new one for systemd?
What are those N ways? Certainly, syslog doesn't support field-structured logging. Protocols like GELF are asynchronous and lossy by design. Java's logging tools have obvious lack of usability on many embedded systems. Tools like Flume are designed to harvest logs off disk after they've already been written.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 1:16 UTC (Tue) by HelloWorld (guest, #56129) [Link]
dlang has been spreading falsehoods and refused to educate himself about the design rationale of systemd, the journal etc. so many times that I suggest you stop wasting your time.
Haters gonna hate, that's life.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 1:30 UTC (Tue) by mgb (guest, #3226) [Link]
Field structured logging might be useful for some people.
So write a field-structured logger and allow people to use it. That's the UNIX way. That's the way that allows UNIX to progress.
Don't leverage your control of pid 1 to force people to use your logger. That's the Poettering way. That's the way that stifles UNIX progress.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 2:06 UTC (Tue) by raven667 (subscriber, #5198) [Link]
> Field structured logging might be useful for some people.
True
> So write a field-structured logger and allow people to use it. That's the UNIX way. That's the way that allows UNIX to progress.
True, and that well describes the systemd approach
> Don't leverage your control of pid 1 to force people to use your logger. That's the Poettering way. That's the way that stifles UNIX progress.
False! I don't know how you got here, no one is forced to use the Journal, it does get a copy of the local logs but they go to the normal syslog daemon too and are written out normally or sent across the network normally.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 2:34 UTC (Tue) by mgb (guest, #3226) [Link]
See e.g. http://rpmfind.net//linux/RPM/fedora/18/i386/s/systemd-19...
systemd REQUIRES libsystemd-journal.so.0
You are Lennart Poettering and I claim my £5.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 3:59 UTC (Tue) by raven667 (subscriber, #5198) [Link]
I am not Lennart Poettering but I'm flattered you think so 8-) No £5 for you. Pro tip: Poettering is mezcalero on LWN.
In any event I didn't dispute that systemd logs to the Journal, just that _YOU_ aren't required to use it because everything still goes to traditional syslog. There is no credible way you could claim that anything is being stifled when compatibility is such an obvious priority.
I'm not sure we are really communicating effectively because you don't seem to be understanding what I am saying and your responses don't make any sense to me. I will assume that you are operating in good faith, bid you a good day and thank you for your time, I don't have anything else to say.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 11:17 UTC (Tue) by nix (subscriber, #2304) [Link]
To be honest, if the default journal is an in-memory passthrough, I don't see why anyone would care, anymore than anyone says that syslog-ng has an 'extra useless log' because it does in-memory buffering to keep disk I/O down.
If one said that systemd had an *optional* field-structured message log, would this suddenly make this problem go away, even though the only difference is to change terminology and declare the journal 'off' rather than 'in-memory'?
(The only thing I really disliked about the journal was the explicit lack of definition of the binary journal format. Since this has now been rectified, my complaints about it are gone.)
you do NOT need to write all your programs together to make them work together. Posted Jan 31, 2013 19:07 UTC (Thu) by sorpigal (subscriber, #36106) [Link]
> If one said that systemd had an *optional* field-structured message log, would this suddenly make this problem go away, even though the only difference is to change terminology and declare the journal 'off' rather than 'in-memory'?
I suspect a great deal of the push-back against systemd comes from this very thing. Given the option of talking about it in a way that doesn't rile people up and talking about it in a way that does, the latter is what happens. If the story were "systemd internally uses its own field-structured log format but still outputs logs to syslog by default (or you can just use the internal format directly)" then you'd see more acceptance, but the story was "systemd has replaced syslog with a new, undocumented, non-text log format." If the goal wasn't to annoy people then I can only say that there exists an incredible natural talent for annoyance which is being wasted in the field of software engineering.
you do NOT need to write all your programs together to make them work together. Posted Jan 31, 2013 19:41 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
remember, this isn't just systemd's internal logs, systemd intercepts logs that the apps write to syslog and turn them into this new structure as well.
you do NOT need to write all your programs together to make them work together. Posted Jan 31, 2013 20:42 UTC (Thu) by davidstrauss (subscriber, #85867) [Link]
> remember, this isn't just systemd's internal logs, systemd intercepts logs that the apps write to syslog and turn them into this new structure as well.
Just keep omitting that systemd's journal forwards the entries *unaltered as syslog*, and you might convince some people that the journal breaks their traditional syslog toolchain.
you do NOT need to write all your programs together to make them work together. Posted Jan 31, 2013 20:46 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
look at the context, I am replying to someone explaining how presenting how the journal deals with logs differently could avoid antagonizing people. I am just pointing out that his statement is not complete, it's not just the internal journald logs that are involved.
you do NOT need to write all your programs together to make them work together. Posted Feb 1, 2013 1:21 UTC (Fri) by davidstrauss (subscriber, #85867) [Link]
> look at the context, I am replying to someone explaining how presenting how the journal deals with logs differently could avoid antagonizing people. I am just pointing out that his statement is not complete, it's not just the internal journald logs that are involved.
I'm aware of the context. I just think it's disingenuous to use how the systemd's journal maintains its own data structures to imply some change happens that creates complexity for syslog users.
Use of internal data structures is true of any log daemon, including rsyslog. Saying that the journal "turn[s] them into this new structure as well" implies that the syslog messages emitted by the journal are not identical to the messages sent in, which is false.
The journal goes beyond syslog, but it does not get in its way or force users to change any existing applications, monitoring, or workflows.
When we updated our configurations for Fedora 17 (the first release with the journal), we didn't have to touch anything to continue getting the same syslog messages we got before in the same places we had always looked for them.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 7:59 UTC (Tue) by smurf (subscriber, #17840) [Link]
Oh come on.
So you want to force the systemd people to support writing two log streams? One for traditional syslog and one for field-structured logging? In what way does that fit your UNIXy "do one job well" definition?
Instead, systemd spits out a nice structured log, journald reads that and translates it to "standard" syslog, other programs can be switched over to structured logging once journald can be assumed to be available (or they can support both if they want the compatibility, or somebody can write a compatibility library with a journal API and syslog output), everybody is happy.
Why the hell should it matter that systemd and journald are maintained in a single repository?
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 17:35 UTC (Tue) by raven667 (subscriber, #5198) [Link]
I'm guessing that most software never will switch to using the Journal and will continue using standard syslog but that's fine and is supported.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 20:43 UTC (Tue) by khim (subscriber, #9252) [Link] Yes, it is supported but in sane manner: systemd only creates log in journald format and journald converts them. Somehow people who like to raise racket about "monolithic design" and "lack of modularity" simultaneously find this solution problematic and want to stuff syslogd format support in systemd itself. Talk about consistency.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 12:07 UTC (Tue) by khim (subscriber, #9252) [Link] you make programs work together, not by writing them together, but by defining the interface between them. That way the different programs can be written by different people at different times, replaced one by one, etc That's nice theory but it does not work in practice, sadly. That is the way to have a long-term robust solution. It does not work this way. Either stuff is developed in lockstep (to use latest version of GLibC you need latest version of GCC and to use latest version of GCC you need not-yet-released-version of binutils), or there are bunch of problems when one component or another does not adhere to the specs. And if you develop bunch of small programs in lockstep then why don't develop them together in a single repository? That's how UNIX was traditionally developed after all! If you consider that all the programs need to be written together, it's a very small step to consider that you can break all your old interfaces because you ship all new versions of everything. Yup. That's the point: if the interfaces only tie these programs together and are not exposed outside then why do you need to keep them static? Practice shows that it's impossible to keep bazillion interfaces robust and stable. Only two possibilities can be realized by real people in real life: APIs need to be defined, and you need to support old ones going forward. Yes. And this is only possible if scope of these APIs are kept limited. you do NOT need to write all your programs together If you want stable ABIs? Yes, you do. It's impossible to keep all the ABIs stable which means that either you have some ABIs "set in stone" and most in "we don't care because these are iternal ABIs" state or you have lots of ABIs in "permanently broken" state.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 12:43 UTC (Tue) by mgb (guest, #3226) [Link] @khim - Software engineers have been doing what you consider impossible for many decades. This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.Doug McIlroy Text interfaces are more resilient, more flexible, and more readily debuggable than binary interfaces. Inappropriate binary interfaces are a hallmark of the premature optimizer. Premature optimization is the root of all evil. Donald Knuth
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 13:44 UTC (Tue) by khim (subscriber, #9252) [Link] Software engineers have been doing what you consider impossible for many decades. Nope. They claimed they do that, sure. But did? Examples, please. Some popular thing which is built a way you claim it should be built. I know of many things which claim to be built from tiny independent components in a fashion you preach. I also know that any small change tend to get a lukewarm response and request to stop using non-standard components. And if we can only use one particular version of component A and one particular version of component B with one particular version of component C then why not develop them together? This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. Nice quote. Yet somehow exactly zero SQL databases (and Unix was big in SQL world till it died) use text interfaces. Says something about applicability of said principle, isn't it? Text interfaces are more resilient, more flexible, and more readily debuggable than binary interfaces. Inappropriate binary interfaces are a hallmark of the premature optimizer. Premature optimization is the root of all evil. Nice, but incomplete quote. The full one says something different: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil" and when compounded by a different quite from the same guy (In established engineering disciplines a 12 % improvement, easily obtained, is never considered marginal and I believe the same viewpoint should prevail in software engineering) nicely explains why journald makes sense, isn't it? Yes, it makes sense because it's in critical path for many workloads, it speedups things by more then 12% and thus it fits nicely in this 3% niche. If you try to counter facts with quotes then at least make sure you understand what the quoted documents say in reality.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 13:59 UTC (Tue) by HelloWorld (guest, #56129) [Link]
The funny thing about this "text streams" meme is that Unix doesn't actually use text streams, yet nobody seems to realise that. A Unix pipe is a byte stream, not a text stream. To interpret a byte stream as text, one needs to negotiate an encoding, and Unix pipes don't offer a means to do that.
And besides, even if Unix did offer text streams, I'd still disagree. Text is not a universal interface, there's a *lot* of data where a text format doesn't make any sense at all, like videos, audio, images, and many many others. And in fact, most programs today don't mess around with text files any longer, but use structured data representations and libraries to (de)serialise them. DBus is a binary protocol because it's faster, but also because nobody cares as everyone uses it via a library (and code generators) anyway.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 17:24 UTC (Tue) by smurf (subscriber, #17840) [Link]
> one needs to negotiate an encoding,
> and Unix pipes don't offer a means to do that.
One might argue that the negotiation is implied by the LOCALE setting.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 18:17 UTC (Tue) by HelloWorld (guest, #56129) [Link]
> One might argue that the negotiation is implied by the LOCALE setting.
The LOCALE setting is just a way to specify manually the information that can't be negotiated through the pipe. If pipes were actually a text stream, there'd be no need to do with manually and things would just work.
Anyway, I don't think such a design would be desirable, because as I said before, text is in fact not a universal interface. Which is of course why many programs today communicate with much more structured protocols like D-Bus.
While I do sympathise with your views about UTF-8, there's a large amount of data stored in legacy encodings, and it's not going away any time soon.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 20:28 UTC (Tue) by anselm (subscriber, #2796) [Link] A Unix pipe is a byte stream, not a text stream. To interpret a byte stream as text, one needs to negotiate an encoding, and Unix pipes don't offer a means to do that. In the interest of fairness it should be mentioned that, at the time Doug McIlroy made the quoted statement, ASCII was still the text encoding of choice (at least if you were in the US). The idea that an encoding might have to be »negotiated« before a pipe could be useful didn't really show up on anyone's radar. Also, most of the usual Unix tools assume that their input consists of lines of text separated by newline characters, rather than arbitrary »byte streams«, and generate output to suit this assumption. Note that UTF-8 was carefully defined (by Ken Thompson, no less) to fit the common Unix notion of »text« – up to a point where many Unix tools will do »reasonable« things when fed UTF-8 data even if they do not include explicit support for UTF-8.
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 16:22 UTC (Tue) by nix (subscriber, #2304) [Link] to use latest version of GLibC you need latest version of GCC and to use latest version of GCC you need not-yet-released-version of binutilsNeither of these things are true. What is true is that older versions are likely less tested than newer ones, and there is a fuzzy line who knows how far back (probably less far than configure tests for) where things stop working entirely. But neither of the constraints you suggest are true, not even slightly. You can build the latest glibc with GCC 4.4 and you can build GCC with really rather aged binutils (2.19 or something, maybe even older).
you do NOT need to write all your programs together to make them work together. Posted Jan 29, 2013 20:35 UTC (Tue) by khim (subscriber, #9252) [Link] But neither of the constraints you suggest are true, not even slightly. This may be true right now, today, but that's because there are no need to change anything rapidly (last major architecture was x86-64 and it was introduced years ago). When new platform is introduced you see things like "Binutils 2.16.91 or newer are required" quite regularly. And things often breaks even if these requirements are not written explicitly. We'll see how AArch64 will affect all that - but I doubt it'll be possible to use binutils "2.19 or something, maybe even older" for that. P.S. Note: I'm not saying it's somehow bad that it's done that way. In fact that's the right thing to do. But if GLibC is tightly tied GCC and GCC is tightly tied to binutils then why pretend that they are different projects? They are not: they are parts of larger project: GNU.
you do NOT need to write all your programs together to make them work together. Posted Jan 30, 2013 16:24 UTC (Wed) by nix (subscriber, #2304) [Link]
Well, yes, of *course* if a new architecture is introduced then you have to use a toolchain new enough to support that architecture! That doesn't mean that all projects that know about that architecture are the same project! Nor does it suddenly imply a tight tying among the projects.
Even among the toolchain-related projects, they have some different maintainers (though some in common), different governance structures, different mailing lists, different release schedules, different source trees and not much shared code infrastructure (the only bits I can think of shared between GNU toolchain components are libiberty and libbfd, though the latter is quite sizeable).
you do NOT need to write all your programs together to make them work together. Posted Jan 31, 2013 22:52 UTC (Thu) by khim (subscriber, #9252) [Link] Even among the toolchain-related projects, they have some different maintainers (though some in common), different governance structures, different mailing lists, different release schedules, different source trees and not much shared code infrastructure (the only bits I can think of shared between GNU toolchain components are libiberty and libbfd, though the latter is quite sizeable). This is true but I'm interaction with these people enough to notice that despite all that development happens in all these projects simultaneously. When you need to add, e.g. something like ifunc you need to change GLibC, binutils, and GCC in a lockstep - and this is done by the same people without regard to any other projects. To me it looks more like a large single project which is complicated by artificial division between glibc/binutils/gcc rather then three separate project: you still need to introduce changes in all these places but additionally you need to write code which will detect version skew and disable these features appropriately. The division lies at the different level: in GCC there are few layers (SSA, RTL, etc) and developers who work with different layers know less about other layers then "low-level GCC people" know about binutils and glibc!
you do NOT need to write all your programs together to make them work together. Posted Feb 2, 2013 19:17 UTC (Sat) by nix (subscriber, #2304) [Link]
All you say there is true -- some changes must affect several layers at once. However, a lot of changes affect glibc and the kernel at once, too -- does that mean that glibc and the kernel are the same project? (Note that for quite a long time the founder and co-maintainer of glibc was also one of the maintainers for one of the nastiest parts of the core kernel, ptrace() and signal handling, so even the people-in-common rule is true!)
I don't think they're the same project -- and neither are the various toolchain projects the same project, anymore than gnulib is the same project as coreutils merely because changes to both often happen in synchrony. They are simply projects with *some* close coupling between them -- but even that coupling is optional (you can build GCC and perhaps even glibc with a binutils that doesn't support IFUNC, and nothing goes wrong).
Poettering: The Biggest Myths Posted Jan 29, 2013 9:39 UTC (Tue) by nix (subscriber, #2304) [Link] I like the idea of systemd -- I'm avoiding it for now simply because I'm a stability freak whose boot process is still stuck using BSD-style monolithic shell scripts, which are really simple and easy to understand, and because I'd need some time to migrate when I didn't need my system to do other stuff. But this is just nonsense:reliable monolithic part (GNU components: GCC, GLibC, binutils, coreutils, etc - they are not really supposed to be used with anything else)GCC, binutils and coreutils are insanely portable. They work on virtually everything and (in the case of GCC and binutils) target virtually everything, though the recent switch of GCC to C++ will restrict the set of bootstrap compilers somewhat. Heck, until very recently there were a bunch of targets on which you could run GCC but could not use binutils -- your claim is false on its face. glibc is also portable, in theory -- it's just that all the ported bits have fallen into disuse and disrepair. The portability framework is unlikely to rot anytime soon because it is also the system library for the Hurd, which at that level is quite thoroughly unlike Linux in some ways. (Also, that framework is also used for cross-arch portability, which is definitely not going away now that glibc's maintainers do not have Uli's charming contempt for all non-x86 architectures.)
Poettering: The Biggest Myths Posted Jan 29, 2013 12:49 UTC (Tue) by khim (subscriber, #9252) [Link] glibc is also portable, in theory -- it's just that all the ported bits have fallen into disuse and disrepair. This is nice theory but it's not supported by practice. You see, just recently we've needed to port GLibC to ARM for PNaCl-arm. We've talked about about this project with Roland and eventually the resolution was to add support for NaCl-arm to GCC. If Roland feels it's easier to add support for the new architecture to GCC rather then port GLibC to Clang I tend to think that it's basically impossible to do in practice. The portability framework is unlikely to rot anytime soon because it is also the system library for the Hurd, which at that level is quite thoroughly unlike Linux in some ways. Sure. You can use GLibC with two different kernels (and soon with third one if you'll count NaCl as separate "kernel") but it can only be used with GCC and binutils - and you need quite recent versions of both. Kinda like systemd with udev, journald and others: you can use udev separately but systemd can only work with one particular version of udev and one particular version of journald. Heck, until very recently there were a bunch of targets on which you could run GCC but could not use binutils -- your claim is false on its face. And there are bunch of targets where you must use GNU ld and GNU as and can not use system-provided versions. Basically it's all very simple: GNU tools are supposed to be used with other GNU tools only but where feasible to use system-tools provided by different platforms they do that. This is very explicitly not some nirvana of free-for-all pick-and-choose, but a set of decisions made by sane people: they used system versions of tools where it was harder to port binutils to the new platform and the forced GNU tools in most other cases.
Poettering: The Biggest Myths Posted Jan 29, 2013 16:14 UTC (Tue) by nix (subscriber, #2304) [Link]
One bit of portability you will *not* generally find is portability for lowlevel components like C libraries to different compilers. I am not at all surprised glibc only works with GCC: it's stuffed with GCC-specific stuff, just as its makefiles are very heavily GNU make dependent. (I'm amazed you thought it might have been worth trying to make it work with something else.)
Poettering: The Biggest Myths Posted Jan 29, 2013 20:22 UTC (Tue) by khim (subscriber, #9252) [Link] One bit of portability you will *not* generally find is portability for lowlevel components like C libraries to different compilers. Many C libraries can be compiled with different compilers. Including GLibC 1.x. But version 2.x was redone to consciously use features of GCC - just like systemd was consciously redone to use features of Linux. Somehow it's ok for GLibC but not for systemd? Why?
Poettering: The Biggest Myths Posted Jan 30, 2013 16:21 UTC (Wed) by nix (subscriber, #2304) [Link]
Well, I wasn't one of those arguing that systemd should be portable across OSes: init systems generally aren't, not even the simple ones. I was arguing against your claim that this is universal in other sorts of software: it is not. Lots of software supports portability in various ways, be that compilation with different compilers (binutils, GCC), or running on different platforms (glibc, to an extent)... and huge amounts of software doesn't care e.g. what glibc or binutils or kernel version it was built with. The coupling is not as tight as you suggest.
Poettering: The Biggest Myths Posted Jan 31, 2013 22:43 UTC (Thu) by khim (subscriber, #9252) [Link] I was arguing against your claim that this is universal in other sorts of software. Please remind me where I've made such claim. Lots of software supports portability in various ways, be that compilation with different compilers (binutils, GCC), or running on different platforms (glibc, to an extent)... and huge amounts of software doesn't care e.g. what glibc or binutils or kernel version it was built with. Lots of peripheral software is portable. The core is not. And this is exactly what I've said in the very beginning: Most popular OSes are all almost completely monolithic per this definition: Android, Busybox, BSDs. Only GNU/Linux is weird creature which includes robust and reliable monolithic part (GNU components: GCC, GLibC, binutils, coreutils, etc - they are not really supposed to be used with anything else) and loose and flaky things like sysvinit with it's plethora of weird scripts. Take any non-Linux-based OS you want - and you'll find out that he core is built from a single repo using a single build script. Kernel, libc, some core utilities. The next ring is more portable: BSD make itself may be ported (and was, in fact, ported) to other platforms but you can not use anything else to build, say, FreeBSD's kernel or libc. The outermost ring is the most portable: these are applications which can be ported to other platforms. But a quirk of history made Linux an exception from this rule and, surprise, surprise, this exactly where a lot problems originates: various tiny core components don't cooperate well enough and require plethora if work from distribution makers to somehow create something usable. And even then they often fail. Note that usually you can not even replace these core components: even distributions which formally offer such ability (like Debian or Gentoo) crumble and start misbehaving when you actually try to use such "freedom". The coupling is not as tight as you suggest. The coupling is as tight as I suggest and to me systemd looks like a try to finally bring this approach to desktop and server linux. Note that places where Linux dominates - such as routers and mobile phones - already use this approach. P.S. It's also looks funny to me when people try to say that what systemd developers are doing is somehow against the "UNIX philosophy" when, in fact, they are doing what UNIX did from a day one and what all direct descendants of UNIX are still doing today.
Poettering: The Biggest Myths Posted Feb 2, 2013 19:14 UTC (Sat) by nix (subscriber, #2304) [Link]
I'm not even going to bother arguing with this. If you want to keep arguing that black is white and that GNU binutils, GCC, and the other parts of the toolchain are nonportable, that's fine. You're wrong, and spouting nonsense, but you're allowed to do that.
Poettering: The Biggest Myths Posted Jan 29, 2013 16:18 UTC (Tue) by nix (subscriber, #2304) [Link] This is very explicitly not some nirvana of free-for-all pick-and-choose, but a set of decisions made by sane people: they used system versions of tools where it was harder to port binutils to the new platform and the forced GNU tools in most other cases.This is entirely incorrect. System versions of tools are supported for toolchain building where they are not too broken to work reliably, particularly if it seems likely that people would otherwise need to resort to cross-compilation to get the thing built on that platform. Heck, the horrible bundled K&R HP-UX 10.x C compiler was supported for many many years, despite barely grasping advanced concepts like functions: if your allegations were true it would never have been supported at all. (It is true that, these days, it is much easier to cross-compile from a random Linux system than it ever was before, so tolerance for totally broken crap like that is going down. On the gnulib list, they say "we are not running a software archaeology project", and they are quite right.) Unfortunately there are a lot of broken assemblers, linkers and the like out there. (Oddly, one of the most often broken components is sed. sed! How hard is it to write a working sed, I ask you...)
Poettering: The Biggest Myths Posted Jan 30, 2013 0:48 UTC (Wed) by malor (subscriber, #2973) [Link] (Oddly, one of the most often broken components is sed. sed! How hard is it to write a working sed, I ask you...)Heh, I bet most of the broken implementations started out with that exact thinking..."how hard could this be?"
Poettering: The Biggest Myths Posted Jan 30, 2013 16:30 UTC (Wed) by nix (subscriber, #2304) [Link]
Most of it seemed to be classic Unix thinking, "we can just use a fixed N-character buffer here, nobody will want more than X of these". So you get fun like seds that coredump if you use more than 512 characters in an s// expression's regexp part, and that sort of thing. A crashing sed tends to annoy configure scripts...
Poettering: The Biggest Myths Posted Jan 31, 2013 21:45 UTC (Thu) by dirtyepic (subscriber, #30178) [Link]
>Sure. You can use GLibC with two different kernels (and soon with third one if you'll count NaCl as separate "kernel") but it can only be used with GCC and binutils - and you need quite recent versions of both.
I'm not sure what you define as quite recent, but as of last month the minimum requirements are binutils-2.20 and gcc-4.3, which are both 3+ years old. Before that you could still build glibc with gcc-2.95.3 with a bit of futzing.
Oh, it also requires a kernel later than 2.6.16 (or 3.7 if you want AArch64 support which is obviously a conspiracy) so I guess you'll have to make some room for it under your tightly-knit GNU umbrella.
|
|||||||||||||