| |||||||||||||
GNU Guix launchesGNU Guix launchesPosted Nov 25, 2012 18:30 UTC (Sun) by oever (subscriber, #987)Parent article: GNU Guix launches The package manager is not new. It is Nix, which is also used in NixOS. Nix is a package manager that allows users to install separate package version from the main packages versions. It also allows easy rollbacks of versions. I believe that it also wants to make building packages from sources deterministic; building a particular version of a package should always result in the exact same binary package, regardless of the system on which it is built. This would allow users to verify that a binary package was actually built from the sources it claims to be built from.
(Log in to post comments)
GNU Guix launches Posted Nov 25, 2012 18:51 UTC (Sun) by idupree (subscriber, #71169) [Link]
Yay! I was thinking "why are they duplicating Nix" but, you note, they're using it!
Re: "building a particular version of a package should always result in the exact same binary package, regardless of the system on which it is built",
I wish that were the goal. Last time I checked, derivations were specified by a transformation A -> B where A is fixed but B is a large nigh-unknowable set of possible binary packages.
It's a lot of work to reduce "B" to a set with a single member. Build dates are often embedded in programs. Systems like `make`, indeed, depend on file timestamps. Parallel builds can make products appear in unspecified orders. `uname -a` can return different results; filesystem directory order can differ; CPU instruction support can vary. Luckily, we wouldn't have to make all of these unobservable to build systems; it is enough to patch build-systems to produce a deterministic result. If we miss something for a package, we just get two conflicting builds and can fix the build script.
How practical is that? How many people are trying to do that work?
GNU Guix launches Posted Nov 26, 2012 1:45 UTC (Mon) by jreiser (subscriber, #11027) [Link]
The "Domain Software Engineering Environment (DSEE)" of Apollo Computer Inc (mid to late 1980's or so) handled many of the "free variables" that you point out. It was possible to reconstruct old builds and get bit-for-bit identical results. For instance, the entire software development tool chain [the identity and version of every tool that touched the code] was captured in the manifest for a build.
GNU Guix launches Posted Nov 26, 2012 7:09 UTC (Mon) by oever (subscriber, #987) [Link]
That sounds like bliss! It would be great if we could have a way to do this today. It would really help with security and debugging and could help reduce compile times. If everybody in a development team is using the same tool chain, determined by the checksum for the toolchain, then many of the compiled files could be shared among the team.
It would allow build systems to look at checksum of the output and input files to see if a file is up to date or needs rebuilding. This is much more reliable than using mtimes. A log would need to be kept of which inputs give which outputs, but that seems worth it.
GNU Guix launches Posted Nov 26, 2012 10:33 UTC (Mon) by rossburton (subscriber, #7254) [Link]
The Yocto Project does something like this to speed up distribution compiles, by checksumming all variables, dependencies, and sources. If the hash exists in the cache, a pre-compiled package is extract and used.
GNU Guix launches Posted Nov 26, 2012 12:55 UTC (Mon) by NAR (subscriber, #1313) [Link]
In a previous project a couple of guys did just that. C++ compiling on ridiculously slow Sparc workstations was not fun, so checksums were made from the sources and object files were reused. If I remember it correctly, the speed gained was not that much as we hoped for and sometimes strange things happened - but this was 10 years ago and my memories could be wrong here. Faster CPU and disk was the real solution.
GNU Guix launches Posted Nov 26, 2012 14:01 UTC (Mon) by vonbrand (subscriber, #4458) [Link] ccache to the rescue... almost invisible, works like a charm (only got bitten when crashes left empty .o files lying around, which ccache deemed correct...)
GNU Guix launches Posted Nov 26, 2012 13:59 UTC (Mon) by vonbrand (subscriber, #4458) [Link] I'm sorry, but if you have a "development team" which shares object files, having all of them using the exact same tools should be a cinch, just have each one run the same version of your favorite distribution (modulo "recompile all worlds" each time GCC/coreutils/... version gets bumped). Not really needed most of the time, I re-make some projects I follow with updates, and that doesn't require to recompile everything even if gcc changes. Only rarely do I have to clean beforehand. Even so, compile timestamps get embedded in object (and other) files, perhaps even host names and assorted other trivia. Just take a look at the lengths GCC's build goes to check that stage 2 and 3 builds are the same.
GNU Guix launches Posted Nov 26, 2012 20:02 UTC (Mon) by zooko (subscriber, #2589) [Link]
The Vesta configuration management tool accomplishes some of what you want by requiring all of your build tools and, in general, everything that could affect the build output, to be served by an NFS local mount, so that Vesta can keep track of all the files that the build process looked at while building. Vesta then makes a copy of the current version of each of those files in its revision control history so that you can rebuild the exact same build output in future years. :-)
(Caveat: obviously there are always more ways that this can go wrong, like your new kernel has a different set of bugs than your old kernel, that affect your compiler's behavior. But still, Vesta's approach sounded good to me.)
https://en.wikipedia.org/wiki/Vesta_%28Software_configura...
Back to the direct topic: I'm excited about Guix! Because I love the ideas of determinism, transactional upgrade and rollback, and the other related features. Nix is also associated with another Very Good Idea — your autobuild system should also run the unit tests automatically, unifying "Continuous Integration" (like Buildbot, Jenkins, etc.) with autobuilder.
GNU Guix launches Posted Nov 26, 2012 7:43 UTC (Mon) by Lev (guest, #41433) [Link]
You'd probably be interested in Vesta, which dealt with such issues. See http://www.vestasys.org/ and especially http://www.vestasys.org/why-vesta.html
GNU Guix launches Posted Nov 26, 2012 8:49 UTC (Mon) by oever (subscriber, #987) [Link] Guaranteed repeatability of builds. Builds completely specify everything that affects their outcome, including the exact versions of all source files, as well as compiler versions, library versions, etc. This makes it possible to perform any build you've ever done in the past and be certain that you will get identical results. (This can be a big help with finding and fixing bugs and other QA issues; with Vesta you never have to worry that a bug has been masked rather than fixed by intervening changes, because you can always re-build the exact version that exhibited the problem.)source In the 'modern' way of working this sounds like magic. I've just tried to build Vesta, but unfortunately it fails on my Fedora machine. So for some reason Vesta did not catch on to a larger audience even though a lot of effort was put in: Vesta is a mature system. It is the result of over 10 years of research and development at the Compaq/Digital Systems Research Center, and it was in production use by Compaq's Alpha microprocessor group for over two and a half years. The Alpha group had over 150 active developers at two sites thousands of miles apart, on the east and west coasts of the United States. The group used Vesta to manage builds with as much as 130 MB of source data, each producing 1.5 GB of derived data.source An updated version of Vesta should probably use Git and combine a sha1 for the toolchain with a sha1 for the code version to get identical binaries and identical binary packages with a checkable checksum.
GNU Guix launches Posted Nov 26, 2012 16:00 UTC (Mon) by welinder (guest, #4699) [Link]
> Guaranteed repeatability of builds
That would require an audit of all packages' build system to ensure
Anything using "date" to embed a timestamp anywhere will not be
GNU Guix launches Posted Nov 26, 2012 18:54 UTC (Mon) by oever (subscriber, #987) [Link]
The only timestamps in the build should be ones that come from the inputs: the build tools and the source code. There should be no use of randomness in a build.
The value of knowing exactly where your code come from is huge. Currently there is no easy way to check that a binary packages correspond to source packages.
GNU Guix launches Posted Nov 28, 2012 9:46 UTC (Wed) by oak (subscriber, #2786) [Link]
For example packages going to OBS (OpenSUSE Build Service) are patched to remove such things as it messes up their daily test re-builds. For example:
https://build.opensuse.org/package/view_file?file=inkscap...
Noticing date & time usage in package sources is easy in daily automated builds. Other differentiators taken from the environment are harder to find though, because build machines are pretty identical.
GNU Guix launches Posted Nov 26, 2012 8:59 UTC (Mon) by oever (subscriber, #987) [Link] I have not read it yet, but a text search shows that the author of Nix, Eelco Dolstra, was aware of Vesta when writing his PhD thesis.
GNU Guix launches Posted Nov 26, 2012 9:57 UTC (Mon) by renox (subscriber, #23785) [Link]
> Yay! I was thinking "why are they duplicating Nix" but, you note, they're using it!
But it still isn't clear: What are the differences between Guix and NixOS?
GNU Guix launches Posted Nov 26, 2012 11:25 UTC (Mon) by cmm (guest, #81305) [Link] My understanding is that they forked Nix to replace its configuration language with Guile. Which is, at least in theory, pretty neat, and is probably very much interesting to Ludovic Courtès personally (he is the Guile maintainer, or was last I checked anyway). The practical benefits of Guix over NixOS are less clear to me, but what do I know.
GNU Guix launches Posted Nov 26, 2012 13:11 UTC (Mon) by renox (subscriber, #23785) [Link]
> My understanding is that they forked Nix to replace its configuration language with Guile.
Thanks for the clarification.
GNU Guix launches Posted Nov 26, 2012 13:51 UTC (Mon) by cmm (guest, #81305) [Link] The whole "standardized extension language" thing brings two benefits:
Which is all cool if you need to add an extension language to your program and haven't done so yet. If you already have one and it's not brain-dead — I don't really see much point in replacing it. Nix's configuration language does not seem brain-dead. As for the alternatives: Lua is younger than Guile, I think, and anyway it's basically a Scheme with syntax (and some questionable design choices mixed in, like not requiring you to declare variables. It's amazing how much typo-related pain that is customarily ascribed to the dynamically-typed nature of "scripting" languages is actually the result of this particular brain damage which most of them share...). I don't know how nice Python is as an extension language — but it is not backward-compatible with itself, which in my eyes makes it a joke (yes, I know the whole world is in love with it anyway). This is not GNU advocacy, I'm just trying to illustrate the fact that their extension language policy (as I understand it) is not arbitrary.
GNU Guix launches Posted Nov 26, 2012 14:10 UTC (Mon) by renox (subscriber, #23785) [Link]
> Lua is younger than Guile,
Yet Lua is much more used than Guile..
GNU Guix launches Posted Nov 26, 2012 14:16 UTC (Mon) by cortana (subscriber, #24596) [Link]
OTOH, Lua development is done behind closed doors.
GNU Guix launches Posted Nov 26, 2012 14:20 UTC (Mon) by cmm (guest, #81305) [Link] Yes, Lua is currently more used than Guile. You may want to consider that when Guile just started, everybody wondered "why not TCL?" or "why not Perl?". Later it got more like "why not Python?", and now it's "why not Lua?". The X in "why not X?" keeps changing, and Guile is still there. :) Also, as I said, surface syntax is really the least important thing in an extension/configuration language.
GNU Guix launches Posted Nov 26, 2012 14:33 UTC (Mon) by renox (subscriber, #23785) [Link]
> The X in "why not X?" keeps changing, and Guile is still there. :)
So? Perl or Python are still much, much more used than Guile..
> Also, as I said, surface syntax is really the least important thing in an extension/configuration language.
You said it but it's not necessarily true: it's very likely that the syntax is a big part of the reasons why the Lisp family has so few users..
GNU Guix launches Posted Nov 26, 2012 14:37 UTC (Mon) by cmm (guest, #81305) [Link]
It's not just likely, it's a fact.
Why are we discussing on this infinitely tedious topic, anyway?
GNU Guix launches Posted Nov 26, 2012 16:34 UTC (Mon) by khim (subscriber, #9252) [Link] Because it's quite relevant. When guile was starting it promised choice of languages to use in extending their program. Translators were supposed to make it possible to write in any language of your choice. Kind of like CLR but on top of scheme. This was ambitious goal and it's implementation could have made guile much better then yet-another-language-run-of-the-mill-extension-language. Yet last time I've checked guile was yet-another-version-of-scheme (as obscure and unknown by general public as any other scheme dialect) which made it kind of useless as an extension language.
GNU Guix launches Posted Nov 26, 2012 16:42 UTC (Mon) by cmm (guest, #81305) [Link] Oh, I realize that. Thing is, the GNU folks realize that quite well too. And since they haven't changed their policy yet, what makes anyone think that rehashing the same tedious arguments yet again will have any effect? BTW, I believe Guile comes with a reasonably functional implementation of ECMAScript these days. I don't think anyone is using that, though. Perhaps that's selection bias at work (if you've already chosen Guile over, say, Lua, then it's a safe bet you like Scheme), and/or perhaps syntax is just Not Interesting enough anyway.
GNU Guix launches Posted Nov 26, 2012 17:30 UTC (Mon) by tzafrir (subscriber, #11501) [Link]
I read the name of that language as EMACSscript.
GNU Guix launches Posted Nov 27, 2012 0:09 UTC (Tue) by nix (subscriber, #2304) [Link]
It comes with that too :) at least, it comes with a (partial, last I checked) implementation of emacs lisp.
GNU Guix launches Posted Nov 26, 2012 21:30 UTC (Mon) by khim (subscriber, #9252) [Link] BTW, I believe Guile comes with a reasonably functional implementation of ECMAScript these days. Really? Since when? Manual says ECMAScript was not the first non-Schemey language implemented by Guile, but it was the first implemented for Guile's bytecode compiler. The goal was to support ECMAScript version 3.1, a relatively small language, but the implementor was completely irresponsible and got distracted by other things before finishing the standard library, and even some bits of the syntax. So, ECMAScript does deserve a mention in the manual, but it doesn't deserve an endorsement until its implementation is completed, perhaps by some more responsible hacker. Not a ringing endorsement to say the least. The only language which is kinda-sorta-maybe-supported is Emacs Lisp and it's not clear why anyone will want that if they don't like Scheme. P.S. It's funny, really. Guile is still pushed really hard by FSF backers and it's used in some fringe projects (like Lilypond, or Gnucash, or, I don't know, Guix) but when serious people need real scripting for their real needs they choose... something else. In fact you can easily see when project built around guile tries to finally reach normal people: it's when guile is finally get a sane alternative.
GNU Guix launches Posted Nov 26, 2012 23:09 UTC (Mon) by elanthis (guest, #6227) [Link]
Wow, that manual entry actually insults someone who contributed code for free? Good job on completely pushing away other on-the-fence contributors. Some people do have more to do with their time than try to fully implement a complex feature for nothing in return besides potential contempt and ridicule.
GNU Guix launches Posted Nov 27, 2012 0:10 UTC (Tue) by nix (subscriber, #2304) [Link]
I suspect you'll find the manual entry was written by the very person who got distracted. :) it feels like self-deprecation to me.
GNU Guix launches Posted Nov 27, 2012 1:27 UTC (Tue) by elanthis (guest, #6227) [Link]
Ah, alright. That obviously doesn't translate very well with a quick reading. :)
GNU Guix launches Posted Nov 26, 2012 17:57 UTC (Mon) by vonbrand (subscriber, #4458) [Link] "Uniform surface syntax" among extension languages doesn't matter that much, what matters much more is uniform(ish) syntax/programming model with whatever the user does all day long. Scheme might be the nicest extension language around, but if you program C all day it will be all Greek to you. Plus what "power users" really do is to tweak the configuration a bit, or add some piece of code found on the Internet. They do not waste a significant slice of their time futzing around with configurations/extensions of the tools.
GNU Guix launches Posted Nov 26, 2012 15:18 UTC (Mon) by SEJeff (subscriber, #51588) [Link]
I think this blurb from a mailinglist[1] does a better job describing Guix/Nix than you did:
Guix & Nix
Nix is really two things: a package build tool, implemented by a library
[1] http://lists.gnu.org/archive/html/guile-user/2012-07/msg0...
GNU Guix launches Posted Nov 26, 2012 12:37 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
Because Nix has one fatal flaw - they didn't write it!
|
|||||||||||||