you do NOT need to write all your programs together to make them work together. [LWN.net]

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 12:07 UTC (Tue) by khim (subscriber, #9252)
In reply to: you do NOT need to write all your programs together to make them work together. by dlang
Parent article: Poettering: The Biggest Myths

you make programs work together, not by writing them together, but by defining the interface between them. That way the different programs can be written by different people at different times, replaced one by one, etc

That's nice theory but it does not work in practice, sadly.

That is the way to have a long-term robust solution.

It does not work this way. Either stuff is developed in lockstep (to use latest version of GLibC you need latest version of GCC and to use latest version of GCC you need not-yet-released-version of binutils), or there are bunch of problems when one component or another does not adhere to the specs.

And if you develop bunch of small programs in lockstep then why don't develop them together in a single repository? That's how UNIX was traditionally developed after all!

If you consider that all the programs need to be written together, it's a very small step to consider that you can break all your old interfaces because you ship all new versions of everything.

Yup. That's the point: if the interfaces only tie these programs together and are not exposed outside then why do you need to keep them static?

Practice shows that it's impossible to keep bazillion interfaces robust and stable. Only two possibilities can be realized by real people in real life:
1. Large amount of interfaces are in constant flux but some are declared "sacred" and kept around forever.
2. Large amount of interfaces are declared "stable" yet in practice they all evolve in some way and you need to patch system all the time to keep it running.

APIs need to be defined, and you need to support old ones going forward.

Yes. And this is only possible if scope of these APIs are kept limited.

you do NOT need to write all your programs together

If you want stable ABIs? Yes, you do. It's impossible to keep all the ABIs stable which means that either you have some ABIs "set in stone" and most in "we don't care because these are iternal ABIs" state or you have lots of ABIs in "permanently broken" state.

(Log in to post comments)

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 13:44 UTC (Tue) by khim (subscriber, #9252) [Link]

Software engineers have been doing what you consider impossible for many decades.

Nope. They claimed they do that, sure. But did? Examples, please. Some popular thing which is built a way you claim it should be built.

I know of many things which claim to be built from tiny independent components in a fashion you preach. I also know that any small change tend to get a lukewarm response and request to stop using non-standard components.

And if we can only use one particular version of component A and one particular version of component B with one particular version of component C then why not develop them together?

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
Doug McIlroy

Nice quote. Yet somehow exactly zero SQL databases (and Unix was big in SQL world till it died) use text interfaces. Says something about applicability of said principle, isn't it?

Text interfaces are more resilient, more flexible, and more readily debuggable than binary interfaces. Inappropriate binary interfaces are a hallmark of the premature optimizer.

Premature optimization is the root of all evil.
Donald Knuth

Nice, but incomplete quote. The full one says something different: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil" and when compounded by a different quite from the same guy (In established engineering disciplines a 12 % improvement, easily obtained, is never considered marginal and I believe the same viewpoint should prevail in software engineering) nicely explains why journald makes sense, isn't it? Yes, it makes sense because it's in critical path for many workloads, it speedups things by more then 12% and thus it fits nicely in this 3% niche.

If you try to counter facts with quotes then at least make sure you understand what the quoted documents say in reality.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 13:59 UTC (Tue) by HelloWorld (guest, #56129) [Link]

The funny thing about this "text streams" meme is that Unix doesn't actually use text streams, yet nobody seems to realise that. A Unix pipe is a byte stream, not a text stream. To interpret a byte stream as text, one needs to negotiate an encoding, and Unix pipes don't offer a means to do that.

And besides, even if Unix did offer text streams, I'd still disagree. Text is not a universal interface, there's a *lot* of data where a text format doesn't make any sense at all, like videos, audio, images, and many many others. And in fact, most programs today don't mess around with text files any longer, but use structured data representations and libraries to (de)serialise them. DBus is a binary protocol because it's faster, but also because nobody cares as everyone uses it via a library (and code generators) anyway.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 17:24 UTC (Tue) by smurf (subscriber, #17840) [Link]

> one needs to negotiate an encoding,
> and Unix pipes don't offer a means to do that.

One might argue that the negotiation is implied by the LOCALE setting.
Or that it is not necessary these days, because anybody who does not use UTF-8 deserves to lose. :-P

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 18:17 UTC (Tue) by HelloWorld (guest, #56129) [Link]

> One might argue that the negotiation is implied by the LOCALE setting.
The LOCALE setting is just a way to specify manually the information that can't be negotiated through the pipe. If pipes were actually a text stream, there'd be no need to do with manually and things would just work.

Anyway, I don't think such a design would be desirable, because as I said before, text is in fact not a universal interface. Which is of course why many programs today communicate with much more structured protocols like D-Bus.

While I do sympathise with your views about UTF-8, there's a large amount of data stored in legacy encodings, and it's not going away any time soon.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 20:28 UTC (Tue) by anselm (subscriber, #2796) [Link]

A Unix pipe is a byte stream, not a text stream. To interpret a byte stream as text, one needs to negotiate an encoding, and Unix pipes don't offer a means to do that.

In the interest of fairness it should be mentioned that, at the time Doug McIlroy made the quoted statement, ASCII was still the text encoding of choice (at least if you were in the US). The idea that an encoding might have to be »negotiated« before a pipe could be useful didn't really show up on anyone's radar.

Also, most of the usual Unix tools assume that their input consists of lines of text separated by newline characters, rather than arbitrary »byte streams«, and generate output to suit this assumption. Note that UTF-8 was carefully defined (by Ken Thompson, no less) to fit the common Unix notion of »text« – up to a point where many Unix tools will do »reasonable« things when fed UTF-8 data even if they do not include explicit support for UTF-8.

you do NOT need to write all your programs together to make them work together.

Posted Jan 29, 2013 20:35 UTC (Tue) by khim (subscriber, #9252) [Link]

But neither of the constraints you suggest are true, not even slightly.

This may be true right now, today, but that's because there are no need to change anything rapidly (last major architecture was x86-64 and it was introduced years ago). When new platform is introduced you see things like "Binutils 2.16.91 or newer are required" quite regularly. And things often breaks even if these requirements are not written explicitly.

We'll see how AArch64 will affect all that - but I doubt it'll be possible to use binutils "2.19 or something, maybe even older" for that.

P.S. Note: I'm not saying it's somehow bad that it's done that way. In fact that's the right thing to do. But if GLibC is tightly tied GCC and GCC is tightly tied to binutils then why pretend that they are different projects? They are not: they are parts of larger project: GNU.

you do NOT need to write all your programs together to make them work together.

Posted Jan 30, 2013 16:24 UTC (Wed) by nix (subscriber, #2304) [Link]

Well, yes, of *course* if a new architecture is introduced then you have to use a toolchain new enough to support that architecture! That doesn't mean that all projects that know about that architecture are the same project! Nor does it suddenly imply a tight tying among the projects.

Even among the toolchain-related projects, they have some different maintainers (though some in common), different governance structures, different mailing lists, different release schedules, different source trees and not much shared code infrastructure (the only bits I can think of shared between GNU toolchain components are libiberty and libbfd, though the latter is quite sizeable).

you do NOT need to write all your programs together to make them work together.

Posted Jan 31, 2013 22:52 UTC (Thu) by khim (subscriber, #9252) [Link]

Even among the toolchain-related projects, they have some different maintainers (though some in common), different governance structures, different mailing lists, different release schedules, different source trees and not much shared code infrastructure (the only bits I can think of shared between GNU toolchain components are libiberty and libbfd, though the latter is quite sizeable).

This is true but I'm interaction with these people enough to notice that despite all that development happens in all these projects simultaneously. When you need to add, e.g. something like ifunc you need to change GLibC, binutils, and GCC in a lockstep - and this is done by the same people without regard to any other projects. To me it looks more like a large single project which is complicated by artificial division between glibc/binutils/gcc rather then three separate project: you still need to introduce changes in all these places but additionally you need to write code which will detect version skew and disable these features appropriately.

The division lies at the different level: in GCC there are few layers (SSA, RTL, etc) and developers who work with different layers know less about other layers then "low-level GCC people" know about binutils and glibc!

you do NOT need to write all your programs together to make them work together.

Posted Feb 2, 2013 19:17 UTC (Sat) by nix (subscriber, #2304) [Link]

All you say there is true -- some changes must affect several layers at once. However, a lot of changes affect glibc and the kernel at once, too -- does that mean that glibc and the kernel are the same project? (Note that for quite a long time the founder and co-maintainer of glibc was also one of the maintainers for one of the nastiest parts of the core kernel, ptrace() and signal handling, so even the people-in-common rule is true!)

I don't think they're the same project -- and neither are the various toolchain projects the same project, anymore than gnulib is the same project as coreutils merely because changes to both often happen in synchrony. They are simply projects with *some* close coupling between them -- but even that coupling is optional (you can build GCC and perhaps even glibc with a binutils that doesn't support IFUNC, and nothing goes wrong).