Weekly Edition
Return to the Security page
| |||||||||||||
Van den Oever: Is that really the source code for this software?
At his blog, Jos van den Oever looks into recreating binaries from their published source code to verify that the executable contains what it says it does.
"A license that promises access to the source code is one thing, but an interesting question is: is the published source code the same source code that was used to create the executable? The straightforward way to find this out is to compile the code and check that the result is the same. Unfortunately, the result of compiling the source code depends on many things besides the source code and build scripts such as which compiler was used. No free software license requires that this information is made available and so it would seem that it is a challenge to confirm if the given source code corresponds to the executable."
(Log in to post comments)
Van Den Oever: Is that really the source code for this software? Posted Jun 20, 2013 20:21 UTC (Thu) by ledow (guest, #11753) [Link]
I think the point is missed.
Just a recompile on the system is not enough, you need to know the full build environment.
And if you REALLY want to know that the source generates the binary - don't use the binary. Compile from source yourself and use THAT binary. That's the only way to be sure, really.
However, most people trust the large distros not to do anything nasty, so we're happy to trust their binaries and published hashes for those binaries. If we weren't then, well, we wouldn't even be using their source because it's scarily easy to sneak something into a billion lines of code without any noticing, even if they are a professional code auditor.
If you're that paranoid, you need to be an expert code reviewer in just about every area. The fact is that open-source is secure not because YOU can review everything, because like science no one person can ever know enough any more to know whether EVERYTHING is correct. It's secure because there are many eyes. If a binary exhibited some behaviour and was traced to a mismatch between source and binary, we'd ALL know about it, and trust of those responsible would evaporate overnight (like confidence in certain CA's after misusing their trust).
You can't review that amount of code. You can easily guarantee that the code generates a given binary unless you compile from source yourself or religiously check hashes of binaries generated in an identical build environment. Neither of these is a big problem in the global scheme of things. But the first is certainly INFINITELY more important than the last.
If you cared, you wouldn't be using supplied binaries for anything.
Van Den Oever: Is that really the source code for this software? Posted Jun 20, 2013 20:33 UTC (Thu) by mahrud (guest, #88401) [Link]
A while ago I read about verifiable compiling which is basically providing a way to make sure that the assembly and the source are the same. For instance CompCert (http://compcert.inria.fr/doc/) is a verifiable C compiler.
Quote from the website introduction:
This is a pretty academic topic though.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 20, 2013 21:01 UTC (Thu) by david.a.wheeler (subscriber, #72896) [Link] It requires a lot of information to rebuild the same executable from source, but it's doable. You basically have to convince the builder to (1) record that info and (2) verify that others can rebuild it exactly. There are additional complications if the program is a compiler and it's subverted, because compilers can subvert themselves. This is the "trusting trust" attack that Karger and Schell briefly described in 1974, and that Ken Thompson later demonstrated. If you want to counter the trusting trust attack, take a look at my PhD dissertation, "Fully Countering Trusting Trust through Diverse Double-Compiling", which lets you use a second compiler (possibly in a different environment) as a check on a compiler.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 21, 2013 13:49 UTC (Fri) by paulj (subscriber, #341) [Link]
Aside: While I think your PhD is really interesting work, it's very jarring to hear it described as "_Fully_ countering" the Thompson attack. It just doesn't. Indeed, in that it requires a compiler that is trusted or otherwise verified to compile the compiler correctly it _reinforces_ Thompson's point. I heard the "fully counters Thompson" claim bandied around often enough that I was driven to write up my critique for more carefully: http://pjakma.wordpress.com/2010/09/20/critique-of-divers...
Again, really interesting PhD, but the "Fully" part of the claim made for it just makes my brains throw exceptions. :) The points made by Thompson still fully stand.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 21, 2013 15:56 UTC (Fri) by david.a.wheeler (subscriber, #72896) [Link] I stand by my use of the word "fully". The essence of the "trusting trust" attack is that there's a single point of failure: The compiler executable itself. DDC destroys the single point of failure, because you can now use other compiler executables as a check. Note the plural, too. As pointed out in the dissertation, you can use multiple compilers as checks; an attacker would have to subvert all of them, with the same triggers and payloads, to remain undetected. That changes everything. Instead of blindly picking something we must trust, we can verify components using multiple starting points. That means we can move from "trusting trust" to "trust but verify". I'd rather live in a "trust but verify" world, it's a lot safer. Even if you don't verify, the fact that there is now a mechanism that enables later detection (for a previously-undetectable attack) reduces the risk that someone will try to do it. Applying DDC by itself isn't the same thing as a formal proof that the compiler is okay; if all the compilers had the same triggers and payloads then they'd match. But defenders can make this remarkably hard for an attacker to do. And if you want a formal proof as a basis, use a formally-proved compiler (like CompCert) as your starting point... DDC and formally-proved compilers match up wonderfully. Similarly, DDC only ensures the source and executable match; you still have to review the source code. But people are a whole lot better reviewing source than executables. So yes, I think DDC is a big change from our old situation. In short, it makes verification practical, instead of blindly trusting one executable as your starting point.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 21, 2013 16:04 UTC (Fri) by felixfix (subscriber, #242) [Link]
Requiring multiple compilers, and assuming that not all are subverted, makes the claim "fully" incorrect.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 25, 2013 13:48 UTC (Tue) by dlthomas (guest, #89935) [Link]
"Not all are subverted" is not the requirement. The requirement is "not all are subverted *in precisely the same way*." If there is any difference at all, DDC will detect it and complain. With fresh reimplementations, this is not hugely difficult to assure.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 25, 2013 14:07 UTC (Tue) by paulj (subscriber, #341) [Link]
Except that DDC complains not just on subversion, but also because of bugs and even deliberate features that lead to non-binary-exact reproducible builds. DDC can not distinguish between normal bugs and 'malicious' bugs - those inserted by an evil compiler author/distributor in order to frustrate DDC. Indeed, assume there are malicious compiler writers out there. They do not need to collaberate in order to subvert DDC, all they need to do is insert apparently mundane bugs or features that mean no two compiles reliably build the exact same binary. Just like tcc had.
You will not be able to tell whether they'd be silly bugs or malicious. The evil compiler authors have no need to collude, other than act to frustrate DDC. And you won't know they are evil.
Also, as per Thompson, you need to verify not just the compiler, but large parts of the system. David says DDC could be applied there too. Other people in these comments have noted just how difficult it is to get reliable builds at system levels and how short-scoped they are in time (even if you can get a reliably reproducible build).
Further, even if all your compilers DO build the same binary, you are still trusting the compiler authors have not colluded AND that all compilers have independent authors/distribution-teams. While it may be possible to have more confidence in these assumptions than in assuming 1 compiler is not subverted, these assumptions are still not absolutely, 100% safe.
Thompson's point was that you simply can not escape having to have /some/ degree of trust, UNLESS, you create ALL the underlying system yourself. It still stands.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 25, 2013 18:51 UTC (Tue) by david.a.wheeler (subscriber, #72896) [Link] "Except that DDC complains not just on subversion, but also because of bugs and even deliberate features that lead to non-binary-exact reproducible builds. DDC can not distinguish between normal bugs and 'malicious' bugs..." - that's not a problem, that's a good thing. Who, exactly, wants to release code that's wrong? Why omplain about a test that detects problems? Sure, you then have to track back why there's a defect, but that's true for all tests."You will not be able to tell whether they'd be silly bugs or malicious. The evil compiler authors have no need to collude, other than act to frustrate DDC. And you won't know they are evil." - You have to gather more data than "DDC didn't work", just like any test. Tests generally say "fail" - you have to gather more info to figure out why it failed, and fix it. But silly bugs are clearly different from trusting trust attacks; trusting trust attacks just don't happen by accident. In fact, my dissertation showed an example - I tested a known-malicious program, and the difference clearly showed that the code was malicious. In that case, the code difference started with an addition that read, "when compiling a login program, do XYZ instead". That's a dead giveaway that something malicious is going on; why would compilation functionality depend on THAT? "Other people in these comments have noted just how difficult it is to get reliable builds at system levels and how short-scoped they are in time (even if you can get a reliably reproducible build)." - But some folks do manage. The original article noted that, in the given test case, "The binary package that was built from a Debian source package was not identical to the published binary package, but the differences are limited to timestamps and the build id in the executable". That's easily addressed, and I expect it to be easy to tweak Debian's processes to even eliminate those. Fedora's process is much more different, because no one demanded it... but it's not a matter of being impossible. "Further, even if all your compilers DO build the same binary, you are still trusting the compiler authors have not colluded AND that all compilers have independent authors/distribution-teams. While it may be possible to have more confidence in these assumptions than in assuming 1 compiler is not subverted, these assumptions are still not absolutely, 100% safe." - The Earth is not absolutely, 100% safe. No individual is absolutely, 100% safe. Even formal methods aren't 100% safe because you must always start with some assumption, which might be falsified. You're asking for a level of confidence that we ask of nothing else. You misunderstand the point of "fully". The point of the term "fully" was to "fully" counter the original problem that there was no way to have independent validation of a compiler. Now you can independently verify what you'll be trusting, to whatever degree you believe is enough.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 25, 2013 19:36 UTC (Tue) by paulj (subscriber, #341) [Link]
DDC does not verify. It highlights reproducibility problems. You then need to go investigate those problems to see what is causing them. Most of them will be due to bugs or deliberate features. Indeed, a malicious compiler writer would throw in lots of those, to see if anyone is even trying to do DDC, before ever releasing their malicious subversion. Again, we do *not* live in a world where people are regularly doing DDC of significant parts of the system with *multiple* compilers. Until we do, we do *not* know how easy or difficult it would be. Indeed, if it were easy and DDC provided the verification it promises, why don't we hear more of people doing so?
Next, even if DDC worked for compilers, you're still missing Thompson's point or deliberately taking an artificially restricted view of it. Again, he really did mean the *entire system* - he explicitly mentioned *micro-code* as another level where this attack could be implemented. To fully counter his attack in the abstract requires *full-system* verification.
Even assuming DDC is practical and 100% reliable at the software system level (I've stated why I think it isn't - you disagree): How do you DDC the micro-code? Or the CPU design? Or all the chips in your system?
You bring in Gödel. Thompson's point on trust is as fundamental. Consistent and incomplete, OR complete and inconsistent; re-use the work of others and have to trust it is free of subversion, OR create it all yourself and not have to trust. Imagine a mathematician saying that they've described a complete AND consistent system, for some definition of consistent that isn't quite 100% fully consistent, but hey - nothing is ever fully consistent, is it? Well yeah, that was Gödel's point!
You've described a process, DDC, which you say allows one to re-use the work of others AND not have to trust, for a value of 'not have to trust' that isn't quite 100% free of trust, but hey nothing is ever 100% sure on this earth! Well, yeah... It's still perfectly consistent with Thompson's point - you can't escape trust if you havn't built it all yourself!
Does DDC add something? Quite probably yes! It'd be really interesting to see people try do it regularly (and useful, perhaps - though the rest of us might have to consider how far to trust them ;) ) to evaluate it. But... it's still not countering Thompson's attack, because that attack can not be countered! His point was more fundamental, and abstract, than about compilers subverting compiled binaries.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 25, 2013 20:13 UTC (Tue) by david.a.wheeler (subscriber, #72896) [Link] "DDC does not verify. It highlights reproducibility problems. You then need to go investigate those problems to see what is causing them." - A strange terminology; the same statement is true of a traditional regression test suite. Indeed, any test merely highlights differences between the expected behavior and actual behavior. No test can "verify" in the sense that you seem to imply, since no test, by itself, will tell you in detail why the test failed. Test suites are usually a key part of a "verification" process, so I'm just using the usual meaning of the term (per ISO 15288, etc.). Few people want compilers to incorrectly compile code, or to have malicious code in them; a test that verifies that some of those problems don't exist would typically be part of verification."Most of them will be due to bugs or deliberate features. Indeed, a malicious compiler writer would throw in lots of those, to see if anyone is even trying to do DDC, before ever releasing their malicious subversion. Again, we do *not* live in a world where people are regularly doing DDC of significant parts of the system with *multiple* compilers. Until we do, we do *not* know how easy or difficult it would be. Indeed, if it were easy and DDC provided the verification it promises, why don't we hear more of people doing so?" - Well, this is all pretty recent. I only just released the general dissertation in late 2009 (the 2005 ACSAC paper only discussed a much more restricted case). And a lot of folks have decided that the risks of this attack are low today, and thus aren't spending time to counter it. My hope is that there will be more interest in the years ahead. In any case, as a non-gcc-developer I managed to do it with gcc; someone directly involved in a compiler's development would find it even easier for his compiler. "Next, even if DDC worked for compilers, you're still missing Thompson's point or deliberately taking an artificially restricted view of it. Again, he really did mean the *entire system* - he explicitly mentioned *micro-code* as another level where this attack could be implemented. To fully counter his attack in the abstract requires *full-system* verification. Even assuming DDC is practical and 100% reliable at the software system level (I've stated why I think it isn't - you disagree): How do you DDC the micro-code? Or the CPU design? Or all the chips in your system?" - I discussed this. Section 8.8 talks about lower-level software components like the operating system. Microcode is really just more software, so DDC can still be used (in fact, microcode is often short enough that you can directly inspect it - eliminating the trusting trust attack entirely). Applying DDC to hardware is discussed in section 8.12 of the full dissertation; I discuss it in my presentation too. "You bring in Gödel. Thompson's point on trust is as fundamental. Consistent and incomplete, OR complete and inconsistent; re-use the work of others and have to trust it is free of subversion, OR create it all yourself and not have to trust. Imagine a mathematician saying that they've described a complete AND consistent system, for some definition of consistent that isn't quite 100% fully consistent, but hey - nothing is ever fully consistent, is it? Well yeah, that was Gödel's point!" - There's a third way, trust but verify. Instead of "re-use something and pray it's not subverted", or "do everything yourself", you can "verify something before you reuse it". You can verify it using other compiler(s) that already exist. You can write your own compiler(s), not because you're going to use them, but because your compiler will be used to verify the compiler you will actually use. Note the big difference in the latter case; you are not "creating it all yourself", but instead "creating a local, simpler tool to help verify the real tool". "You've described a process, DDC, which you say allows one to re-use the work of others AND not have to trust, for a value of 'not have to trust' that isn't quite 100% free of trust, but hey nothing is ever 100% sure on this earth! Well, yeah... It's still perfectly consistent with Thompson's point - you can't escape trust if you havn't built it all yourself! - I never said, "you don't have to trust". Clearly you have to eventually trust something. My point is that you can use this technique to verify something before you trust it. Your interpretation of Thompson's paper is different than mine. You seem to think that Thompson's point is that you might have to trust something, and therefore, all is doomed. I think that's not the point at all. Of course you have to trust something. Anyone who uses a bank trusts something. That's boring and not particularly relevant. I think the point of Thompson's paper is that it's difficult to have verification of program-manipulating systems (e.g., compilers). I interpret "trusting trust" as "we must blindly trust something". That's why I say the problem is resolved. We now have a new mechanism for independent verification, one we didn't have before.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 21, 2013 16:24 UTC (Fri) by paulj (subscriber, #341) [Link]
I considered the resistance from attack that diversity in multiple compilers might give, but even in that there are a lot of assumptions of trust. E.g. assuming that different compilers have independent authors, or independent sets of people involved in their release, seems a weak assumption to me.
Further, as Thompson pointed out, the compiler is just one vector, and he explicitly noted his attack could be implemented in _any_ programme-handling programme. Thus, it is not just the compiler, but the whole environment the compiler and any other programme-handling programme run in that must be verified or trusted.
Finally, assume that increasing numbers of additional compilers might buy more re-assurance. That does not mean it is practical though. Even with just *1* compiler, tcc, you ran into issues (bugs) that made reliable builds to the same binary difficult. The more compilers you add to the mix, the more of these problems you run into, and the less practical it becomes.
My critique gives more examples of assumptions that equate to having to write/verify everything yourself OR trust, as was Thompson's point.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 21, 2013 19:02 UTC (Fri) by david.a.wheeler (subscriber, #72896) [Link] A few comments... "assuming that different compilers have independent authors, or independent sets of people involved in their release, seems a weak assumption to me." - Actually, I believe it's pretty strong. With some exceptions, different compilers are written by different people. And if they ARE the same people, well, use a different one, or write your own for use in testing. But you do not need to write a "production-quality compiler" to verify a compiler using DDC, you just need one that functions correctly for one program... and that makes it a much easier problem. "Further, as Thompson pointed out, the compiler is just one vector, and he explicitly noted his attack could be implemented in _any_ programme-handling programme. Thus, it is not just the compiler, but the whole environment the compiler and any other programme-handling programme run in that must be verified or trusted." - Sure, but DDC can used for them, too. You don't have to limit it to just the "compiler" as traditionally defined. I specifically talk about using DDC to test the compiler+operating system as a system. "Finally, assume that increasing numbers of additional compilers might buy more re-assurance. That does not mean it is practical though. Even with just *1* compiler, tcc, you ran into issues (bugs) that made reliable builds to the same binary difficult. The more compilers you add to the mix, the more of these problems you run into, and the less practical it becomes." - Ah, but the problems only normally happen the first time you're applying DDC, and in any case, it's usually hard to do something when no one has ever done it before. "My critique gives more examples of assumptions that equate to having to write/verify everything yourself OR trust, as was Thompson's point." - I think we fundamentally differ on what the problem is, leading us to differences on whether or not I've solved it :-). I believe the problem is having to totally trust any one thing, without being able to verify it. If you can independently verify something, then it's no longer the total "trusting trust" problem that Thompson pointed out. Sure, you have to trust something, but if you can independently verify that thing you trust in multiple ways, it's nowhere near as risky.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 21, 2013 20:40 UTC (Fri) by paulj (subscriber, #341) [Link] Actually, I believe it's pretty strong. With some exceptions, different compilers are written by different people. And if they ARE the same people, well, use a different one, or write your own for use in testing. But you do not need to write a "production-quality compiler" to verify a compiler using DDC, you just need one that functions correctly for one program... and that makes it a much easier problem.I'll be honest, I don't have direct experience of the compiler world. However, in my area of networking, the more specialised you get, the fewer people there are in that speciality. Further, those people move around. They go from one major player to another major player, to start-ups, back to major players; they go from working on open-source stuff to proprietary software and back, and vice-versa. I don't know specifically about compiler writers, but I bet there's a similar flow of people between different organisations and projects there. To me, the notion that different compilers will inherently have independent sets of authors AND other involved engineers (maintenance, release engineering), is highly inconsistent with freedoms which I, and I suspect many others, know engineers/authors avail of in their careers. Further, even if the people were independent, that *still* doesn't mean the software will be untainted by the others. I'm sure some of those authors will at times try out the competing software on their systems at times. I'm just rehashing from my critique, so I get the feeling you mustn't have read it. ;) Ah, but the problems only normally happen the first time you're applying DDC, and in any case, it's usually hard to do something when no one has ever done it before. Agreed, it's possible that you could get agreement among compiler people that exact-reproducibility was important. Sufficient number of them might make their compilers support such a thing. Other parts of the build environment might also have to be adapted too. It's *possible*, but that world doesn't yet exist, it seems. Exactly how easy it would be to get that world AND maintain it, I don't know. I think we fundamentally differ on what the problem is, leading us to differences on whether or not I've solved it :-). Agreed. You've taken an extremely narrow, restricted view of Thompson's attack. That it was specifically about subverting a compiler binary. Even with that restricted view, I believe your technique still requires one to *write* their own compiler from scratch AND/OR trust an existing binary. Further, that restricted view of his attack/point is not right - his paper clearly intended the compiler attack to be an illustration of a much more general class of attacks, in order to make fundamental points about trust. The technique in your thesis is very useful, and seems to me like it could indeed help increase the level of assurance we might have that our software hasn't been subverted. No argument there. It's just that "Fully" qualifier in the claim that's jarring - it's not supported, not even with the deliberately restricted view taken of Thompson's attack. ;) Thompson's point still stands, indeed it is re-inforced by how your technique requires a trusted or verified boot-strap compiler: We must build the whole system ourselves, OR we must trust in those who supplied the parts we did not build.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 21, 2013 22:50 UTC (Fri) by david.a.wheeler (subscriber, #72896) [Link] "I don't know specifically about compiler writers, but I bet there's a similar flow of people between different organisations and projects there.". - Certainly it happens, but it's less common than you might think. Learning how to write a compiler is a standard Computer Science class, and in many cases students actually write a compiler as part of the class. Thus, there are a huge number of people who know how to do it. It's not like there are only 10 people in the world who know how to write compilers. Especially since, for DDC, it only needs to work at all for one program... slow is just fine. "To me, the notion that different compilers will inherently have independent sets of authors AND other involved engineers (maintenance, release engineering), is highly inconsistent with freedoms which I, and I suspect many others, know engineers/authors avail of in their careers.". - Even if there's overlap, the issue is that someone would have to be able to subvert the binary... which means they'd have to have a fair amount of trust, or break in somehow in the build chain, to do it. Typically the "new guy" isn't trusted to the same degree, making it harder to do. "I'm sure some of those authors will at times try out the competing software on their systems at times.". - So? Trying out software doesn't perform the attack. Even if you create a modified malicious compiler on your local system, that means nothing. For an attack to work against DDC, an attacker has to subvert the binaries of multiple compilers, as used by the organization applying DDC. "You've taken an extremely narrow, restricted view of Thompson's attack. That it was specifically about subverting a compiler binary. Even with that restricted view, I believe your technique still requires one to *write* their own compiler from scratch AND/OR trust an existing binary. Further, that restricted view of his attack/point is not right - his paper clearly intended the compiler attack to be an illustration of a much more general class of attacks, in order to make fundamental points about trust." It's not narrow at all. I clearly state in my paper that DDC isn't limited to "compilers" as they are traditionally defined. Indeed, I specifically note that the DDC "compiler" could be an assembler, could include an entire operating system, and could apply to hardware (not just microcode) as well. But as far as what Thompson meant, well, let's look what Thompson says in "Reflections on Trusting Trust". He says that "You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.)" He says his attack is specifically about program-handling code: "I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode". And the reason can't trust others' code is because you're unable to do source-level verification: "No amount of source-level verification or scrutiny will protect you from using untrusted code." The fundamental problem is that with program-handling programs, the creator of the executable can create self-perpetuating attacks. So... let's employ multiple different program-handling programs, make it hard for them to collude, and use them to check each other. Now we're no longer dependent on any single supplier to decide if we get malicious code or not. "The technique in your thesis is very useful, and seems to me like it could indeed help increase the level of assurance we might have that our software hasn't been subverted. No argument there. It's just that "Fully" qualifier in the claim that's jarring - it's not supported, not even with the deliberately restricted view taken of Thompson's attack. ;) Thompson's point still stands, indeed it is re-inforced by how your technique requires a trusted or verified boot-strap compiler: We must build the whole system ourselves, OR we must trust in those who supplied the parts we did not build." You keep ignoring a third way - use multiple parties to check each other. One of those parties could even be you! Outside of the computing field this use of multiple checks is the normal way of gaining trust. Instead of blindly trusting business B, we might get an independent auditor to check out the books. If we're really concerned, we'll get two auditors and make sure that the external organizations that manage B's accounts actually hold the money claimed. Historically, compilers and other program-handling programs were like a business without auditing; we had to trust whatever it produced. DDC creates an auditing step; sure, it's possible to fool it through collusion, but there are many well-known ways to make collusion hard. Moving from blind trust to an auditable process is a big step forward. Put simply: The problem, as I see it, is the previous impracticality of independent verification of program-handling programs. DDC fully eliminates that problem, because it does provide a mechanism for independent verification. Now the problem changes to "how much verification is enough" instead of "I guess we just hope it's okay." How and when do you apply that mechanism? Cue Neo's voice: "Where we go from there is a choice I leave to you."
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 21, 2013 23:18 UTC (Fri) by paulj (subscriber, #341) [Link]
Some quick replies:
- I would not argue few people are _capable_ of writing compilers. Rather, reality is that, relative to the number of developers in software, few people will end up working on widely used compilers. This is just an artifact of the world, the breadth of complexity of our systems, and the need for specialisation. Many people _could_ work on compilers, _few_ do.
- If a compiler developer tests out the binary compiler of another team on his system, that compiler binary could "infect" binaries of his own compiler. As per my critique, to fully counter Thompson's attack (unless you limit his attack in a way Thompson did not intend), you must _at least_ fully solve the problem of virus detection. DDC does not do this, AFAICT.
- “So... let's employ multiple different program-handling programs, make it hard for them to collude, and use them to check each other. Now we're no longer dependent on any single supplier to decide if we get malicious code or not.”
You've gone from trust in 1 compiler, to trust in multiple compilers and assuming any subversion of them is independent. You are clearly _still_ having to use trust (Thompson's point). Further, there are at least several assumptions you're relying on that are unsafe (independence of compiler writers; collusion of such writers as the _only_ way compilers could be subverted).
- “the problem changes to "how much verification is enough"”, if you have fully countered something, why is there a grey area of how much verification is needed?! So, the attack is fully countered, but I now have to place trust instead in having used enough compilers? :)
Anyway, really useful technique that could improve levels of assurance, yes! "Fully" countered? Small, but important point: Not at all. Your work _re-inforces_ Thompson's point! :)
Also, note that your experience with tcc suggests there Thompson attackers could deploy counter-measures against the DCC technique which would be indistinguishable from ordinary bugs!
Fully = no longer a single point of trust Posted Jun 22, 2013 20:10 UTC (Sat) by david.a.wheeler (subscriber, #72896) [Link] "You've gone from trust in 1 compiler, to trust in multiple compilers and assuming any subversion of them is independent." - No. The point is that subverting a SET of compilers is much harder than subverting one. It's not that an attacker could subvert one, it's that the attacker HAS to subvert them ALL. If ANY of the compilers (original + all DDC compilers) is not subverted the same way, then the trusting trust attack is revealed. That changes everything. Instead of a single point of failure, or multiple single points of failure, the attacker must now simultaneously succeed in subverting all of the compilers used in DDC (as well as the original compiler under test). What's worse, the attacker will often not know what to compilers attack, and a defender can even create a special compiler just for DDC (which is much easier to create compared to a general-purpose compiler). "You are clearly _still_ having to use trust (Thompson's point)." - No, I don't think that's the point. Thompson's paper isn't titled "Reflections on trust"! The problem isn't trust, it's "trusting trust" - that is, the need to totally trust some executable since there is no way to independently verify it. Which is why Thompson says you have to write everything yourself. Now we have a way to independently verify program-handling programs (like compilers); we no longer need to continue "trusting trust". DDC fully counters the trusting trust problem, because it provides a mechanism to independently verify the compiler under test. DDC allows you to decide what to use as your independent tests, and of course, their quality will determine DDC's effectiveness. But that's okay, because the defender gets to choose. You could even write your own compiler (call it "Z") for use in DDC, but notice that this is radically different than Thomson's assumptions. You could use Z to check the real compiler, using DDC, and then use the "real" compiler for everything else. Thompson assumes that, in order to trust it, you must write everything yourself. DDC squelches that; you might choose to write the verification tools yourself, and then use someone else's tools once you've verified them. That is quite different, in a very fundamental way.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 22, 2013 20:20 UTC (Sat) by david.a.wheeler (subscriber, #72896) [Link] "Many people _could_ work on compilers, _few_ do." - Even fewer work on all of the compilers for a given language, once there is more than one in simultaneous development. "If a compiler developer tests out the binary compiler of another team on his system, that compiler binary could "infect" binaries of his own compiler." - Actually, that's quite unlikely. First of all, the delivered executables for widely-used compilers are typically generated in a compilation farm, not on some individual's desktop. And even on a desktop, it's way easier for a compiler-writer to write code to subvert themselves than to subvert other compilers, especially if the other compiler isn't widely used. "As per my critique, to fully counter Thompson's attack (unless you limit his attack in a way Thompson did not intend), you must _at least_ fully solve the problem of virus detection. DDC does not do this, AFAICT." - Typically viruses won't have access to the build farms used by serious compilers, and it's much more difficult to write a virus to infect a program you have never seen. "Anyway, really useful technique that could improve levels of assurance, yes! "Fully" countered? Small, but important point: Not at all. Your work _re-inforces_ Thompson's point! :)" - I disagree. We no longer need to keep "trusting trust", that is, trust some executable because we have no independent verification technique. "Also, note that your experience with tcc suggests there Thompson attackers could deploy counter-measures against the DCC technique which would be indistinguishable from ordinary bugs." - No. Exactly backwards. The DDC process is so sensitive that it not only detects trusting trust attacks... it also detects some ordinary bugs. Thus, compiler authors should use DDC, because it counters trusting trust attacks and can detect some other kinds of bugs too.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 24, 2013 19:13 UTC (Mon) by david.a.wheeler (subscriber, #72896) [Link]
Quick correction: I said earlier, "it's much more difficult to write a virus to infect a program you have never seen." What I meant was that it's more difficult to write a virus to subtly change the inner semantics of a program you've never seen. Sure, a virus can include a payload and run it, e.g., to copy itself into new executables, or copy all the files to some external site. But compilers typically parse inputs into some internal data model, transform the data model, and then translate that out to assembler or machine code. If you don't know anything about the internal data model, it's hard to write code to subtly change the internal model.
Diverse Double-Compiling (DDC) counters "trusting trust" Posted Jun 22, 2013 23:09 UTC (Sat) by landley (guest, #6789) [Link]
There are plenty of people who do binary analysis. The ones I follow on twitter (such as @0xabad1dea) work for a company called Veracode that does that for a living. They reverse engineer your binary and tell you how to exploit it from a security perspective, reverse engineer trojans people upload, and so on. Even when dealing with a zero day exploit payload, they can have results back the same day.
The GPL is dying for a number of reasons (some of which I covered starting around 15 minutes into my ELC talk, this link should go direct: http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=15m09s ). But some reasons I _didn't_ talk about in that (it wasn't the focus of the talk) include that it's often actually easier to reverse engineer binaries than take crappy vendor code and do something useful with it. (The forcedeth and noveau drivers were for nvidia, but before AMD bought ATI and released specs there were plenty of reverse engineered drivers for that, plus the ARM mali stuff is reverse engineered... there's tons of it.)
Even if you try to obfuscate your binary, there's an entire group of people who unobfuscate stuff for a living, with elaborate techniques and tools and years of experience at it. They're called security researchers, and they have entire conferences (such as defcon) devoted to it.
Rob
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 5:05 UTC (Fri) by brooksmoses (subscriber, #88422) [Link]
Also note that this is not a 1-to-1 relationship. There are many valid assembly codes that could be generated from a given source program -- consider, for example, different levels of compiler optimization. Likewise, there are many source programs that could generate a given assembly code; consider, for example, factoring out something into an inlined function.
Thus, although you can prove the relationship is valid, this does not guarantee repeatability or reversibility.
Van Den Oever: Is that really the source code for this software? Posted Jun 20, 2013 21:18 UTC (Thu) by armijn (subscriber, #3653) [Link]
It's not an unsolvable problem though and quite some research has been done in this area.
The main motivation behind Nix and NixOS was to able to recreate the exact build environment.
At our lab at Utrecht University we had a pile of PCs to do builds of several packages (mostly program transformation systems and compilers). We quickly found out that many daily build systems were too fragile and builds were not reliable enough exactly because not enough of the environment was known and/or fixed, which did not help fixing subtle bugs. This is why Nix tracks a LOT more about the environment.
Going the other way around (finding out the provenance of a binary) is quite a bit trickier but that was not the point of Jos' blogpost :-)
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 14:12 UTC (Fri) by mikachu (guest, #5333) [Link]
For things like gpl2 binaries on locked down platforms, running your own binaries isn't an option, but it would still be nice to know you have the actual source of the code running.
Van Den Oever: Is that really the source code for this software? Posted Jun 20, 2013 20:28 UTC (Thu) by jimparis (subscriber, #38647) [Link] The recent article on Tor Browser Bundle introduced me to Gitian:Gitian is a secure source-control oriented software distribution method. This means you can download trusted binaries that are verified by multiple builders. Gitian uses a deterministic build process to allow multiple builders to create identical binaries.Which seems like a much more serious approach to the problem than simply rebuilding source packages and comparing the output.
Van Den Oever: Is that really the source code for this software? Posted Jun 20, 2013 21:26 UTC (Thu) by Tobu (subscriber, #24111) [Link] gitian relies on virtualisation (qemu+kvm or lxc), but it does very little to make the build deterministic, instead it verifies it by running on multiple hosts. The toolchain has plenty of opportunities to break things (through threads, leaking system time, pids or other properties of the machine), which build scripts have to tune or strip. There is a native-client implementation of pure functions I can't find again, it's a bit hardcore (too minimal to run a build), but virtualising some of the leakable properties might be a workable approach.
Gitian for Fedora Posted Jun 24, 2013 15:53 UTC (Mon) by wtogami (subscriber, #32325) [Link]
http://wtogami.blogspot.com/2013/05/gitian-for-fedora.html
I ported Debian's packages needed by Gitian, so you can now run Gitian kvm builders on Fedora hosts. This is needed to build deterministic binaries like for Bitcoin.
Van Den Oever: Is that really the source code for this software? Posted Jun 20, 2013 20:42 UTC (Thu) by l0b0 (subscriber, #80670) [Link]
A colleague has recently managed to reliably compile multiple in-house and third party open source software packages into binary identical tarball files. If that is possible, it should be feasible to provide enough context for anyone to reproduce the build of a modern package (environment variables, installed packages etc., or perhaps simply a packaged VM).
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 5:32 UTC (Fri) by brooksmoses (subscriber, #88422) [Link]
In theory, it is possible, yes.
One of the things that's hard here is that this is a place where "enterprise" is not a desktop environment. Even at the small software company where I used to work (CodeSourcery, building GCC), we didn't have a single-computer build system. There were common tools directories -- an attempt to get around dependence on the build server distribution -- that were in a specific place and contained locally-patched versions of things for various historic reasons. There were parts of the build process that would send stuff off to another server to be run, and pieces that verified the sources against our particular version control setup. And all of this interconnected to almost everything in our server farm, all tuned through several years of evolution and accretion.
So, yes, in theory you could replicate all that, if you had documentation for exactly how it was all assembled. But it's not something that fits into a VM.
And, of course, it's a recursive problem. Those build tools came from somewhere, and either you're trusting a binary copy (which you'd have to get from the people who made the build system) or you're recompiling them and need to know how they were built, and so on back into the generations.
This is not to say that it's impossible, or that it's not worthwhile; there are plenty of reasons why repeatable non-system-dependent builds have practical use beyond outside verification. I just want to point out that solving the problem on a simple single-computer build system is really only a start at solving the problem for the way most distributed software is built.
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 5:49 UTC (Fri) by brooksmoses (subscriber, #88422) [Link]
Actually, let me follow up to my own post here, and clarify a bit of the point I'm trying to make.
It's certainly possible to get identical binaries by duplicating every piece of the build system -- tool paths, tool versions, build paths, libraries, and so forth. This is not, in practice, a useful solution for software compiled on a large scale (e.g., by a distributor or a commercial provider); either the precisely-specified build system is too simplistic to fit the needs of being part of a large software-building system, or it's too complex to be readily replicable. And then there's the issue of bugfixing in the build system, which means a potential explosion of versions.
It is also possible to work on solving this problem by reducing the set of things that need to be duplicated in order to achieve identical output, and carefully constraining (in a traceable way) the system dependencies. This is a vastly more useful result -- but it is far harder to achieve, because it means working with the compiler and other build tools that are actually used in real-world production, and fixing bugs and misfeatures (which, in other light, may well not be _mis_features), and finding ways to test this sort of thing.
The latter is immensely valuable. If an organization can be sure they will get the same exact build results on (say) an Ubuntu 10.04LTS or 12.04LTS server, that takes a lot of pain out of upgrading build servers. If we can go a few steps farther, and they can be sure they'll get the same exact build results on that build server or on the desktop of a programmer running Windows, that's _really_ powerful, because that removes a bunch of variables from debugging.
And it also means there's actually a reasonable chance of fitting it into a real-world enterprise build environment, such that outsiders could verify it. :)
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 6:25 UTC (Fri) by alison (✭ supporter ✭, #63752) [Link]
Brooks, don't you agree that the right solution is to supply not only the source code but an also image (ISO, qcow, etc.) that can be used to spawn a VM with pre-installed and configured tools to build it? With such a setup, the user who recompiles the code should be able to produce a binary with the same checksum as a distributor of the software.
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 8:21 UTC (Fri) by knan (subscriber, #3940) [Link]
What then makes you sure the provided VM toolchain isn't doing something evil and linking in binary blobs not in the source code?
Trusting trust (and turtles), all the way down.
Van Den Oever: Is that really the source code for this software? Posted Jun 22, 2013 1:07 UTC (Sat) by alison (✭ supporter ✭, #63752) [Link]
knan, I was trying to solve the problem of how we can build a kernel that we can verify to be identical to Upstream's version. The verification is useful in the event of conflicting test results: if my kernel has the same CRC as theirs, they're the same no matter what anyone claims. The pertinent binary blobs come from the chip vendor, whom I'm afraid we are forced to trust!
Van Den Oever: Is that really the source code for this software? Posted Jun 22, 2013 20:37 UTC (Sat) by knan (subscriber, #3940) [Link]
Yes, but then we're back to the premise - you can verify what the compiler spits out is equal to the shipped binary, but you can't verify that the binary does what the source says - since the shipped compiler can be evil.
Van Den Oever: Is that really the source code for this software? Posted Jun 22, 2013 23:27 UTC (Sat) by alison (✭ supporter ✭, #63752) [Link]
knan, I don't share your concern here: I was trying to solve a communication problem, not a trust problem.
For potentially-evil compilers, use Diverse Double-Compiling (DDC) Posted Jun 24, 2013 19:21 UTC (Mon) by david.a.wheeler (subscriber, #72896) [Link]
If your worry is potentially-evil compilers, then use
<a href="http://www.dwheeler.com/trusting-trust/">Diverse Double-Compiling (DDC)</a>, which can counter them by using a set of one or more independent compilers (as a check on the compiler under test). The "compiler" could be not just the program normally called the compiler, but even the entire operating system used to process the programs. See the presentation and paper for details.
Van Den Oever: Is that really the source code for this software? Posted Jun 25, 2013 7:35 UTC (Tue) by brooksmoses (subscriber, #88422) [Link]
No, I don't agree, for two reasons. Specifically, what I disagree with is that it's a solution that is particularly viable or valuable.
First, you have a recursive problem; that image is also software (much of it likely GPL'ed) -- does that mean we also need to supply another image with which that image was built?
Second, you're now providing the requirement that the person building the software use a build environment that could plausibly be packaged up into an image. This adds at least one layer of complexity (and probably several) to the build process, and that has some pretty significant real costs in programmer time -- but it's not at all clear that it has _real_ added value to anyone paying those costs.
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 2:37 UTC (Fri) by fedtroll (guest, #91489) [Link]
I know Jos van den Oever is on the right track. I recently ran Wireshark while I connected to my ISP, and watched Firefox 21 transmit unknown SSL? to Google. I found out "Google Safe Browsing" is a Firefox 21 default with no shut-off switch or notification for users. I no longer trust Mozilla or Firefox, nor Google or Microsoft, after the daily NSA horror stories.
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 4:30 UTC (Fri) by Cato (subscriber, #7643) [Link]
Instructions to disable Google Safe Browsing in Firefox are easily found: http://support.mozilla.org/en-US/questions/922449
This is a security feature to stop people visiting attack sites that deliver malware, and phishing sites - mostly to Windows clients, but it's relevant to Linux servers which are unfortunately sometimes compromised to deliver such malicious content.
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 7:11 UTC (Fri) by dtlin (✭ supporter ✭, #36537) [Link] The URLs you browse to are not sent – only their hashes, and the set of known-bad hashes can be cached locally to avoid server round-trips. The client code is at mozilla-central/source/toolkit/components/url-classifier/; I'm not sure how much it implements the latter, but by default (configurable by urlclassifier.gethashnoise) Firefox sends many random junk hashes on every request, to obscure the real URL. (I've heard Google engineers working on the backend service complain about how much more expensive it is to serve Firefox than Chrome for this reason.)
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 5:18 UTC (Fri) by nim-nim (subscriber, #34454) [Link]
mock will rebuild against the latest package versions in the repository. Since Fedora changes a lot, they will almost certainly be slightly different than the distro state that existed at the time the original package was built, causing slightly different offsets.
There is not good solution to this problem, either you forbid bugs in software so the build environment does not change over time, or you always rebuild all leaves when a dep is changed (increasing the package churn)
Fedora/mock situation can be fixed Posted Jun 21, 2013 23:05 UTC (Fri) by david.a.wheeler (subscriber, #72896) [Link]
You're right about the problem, but there are various ways to fix it.
A trivial way is to always build executables for version X using the development tools from version X-1.
Another way is to rebuild the binaries with the "latest" build tools, twice, bottom up before release. There other good reasons to do this, e.g., this ensures that the final code uses all the latest optimizations. You'd want to do this in stages, of course, and give yourself enough time for testing.
Making it possible to ensure that you can regenerate the executables, given the source code, can eliminate a lot of potential attacks and concerns.
Fedora/mock situation can be fixed Posted Jun 22, 2013 8:10 UTC (Sat) by nim-nim (subscriber, #34454) [Link]
Building with tools with version X-1 would negate all enhancements brought by X tools. That's not only perf enhancements, sometimes the system admin tools depend on build changes provided by new tools. Besides that's only pushing the problem X-1 tools get updates too.
Most packages are not standalone and depend on other packages (not just new tools) to build. To get your perfect builds you need to freeze all libs your software depends on (not just gcc).
The end result is that it is essentially impossible to get reproducible builds of a large software environment without either freezing it (and accepting bugs, including security bugs, won't be fixed) or rebuilding everything all the time (and then you get a build anyone can reproduce for the few days it will live before being replaced by a new build).
Even getting a bootstrapable distro (something that can be built from scratch once, without depending on binaries of suspect origin) is non-trivial due to how many software bits accumulated cyclic dependencies over time. When A depends on B that depends on C which depends on A what is the build order exactly you are supposed to follow so people can reproduce your stack ?
Right now reproducible builds of a whole system is a fantasy, it can only be envisioned for a few leaf packages (or group of packages) ignoring the stack they build on can not be strictly reproduced.
Fedora/mock situation can be fixed Posted Jun 25, 2013 18:30 UTC (Tue) by david.a.wheeler (subscriber, #72896) [Link] "The end result is that it is essentially impossible to get reproducible builds of a large software environment without either freezing it (and accepting bugs, including security bugs, won't be fixed) or rebuilding everything all the time (and then you get a build anyone can reproduce for the few days it will live before being replaced by a new build)." The good news is that we've got a lot more cycles to throw at the problem, and a lot of this is embarrassingly parallel. You can force rebuilds so that on release everything was built with the current dev tools. You want to do that anyway, because that means that all the packages use the current optimizations, etc. There's no reason you have to wait to the very end to do this; you could recompile lower-level dev tools, then slowly move up the tree. Regression tests can catch a lot, since recompilation of unchanged source code should not be changing functionality at all. "Even getting a bootstrapable distro (something that can be built from scratch once, without depending on binaries of suspect origin) is non-trivial due to how many software bits accumulated cyclic dependencies over time. When A depends on B that depends on C which depends on A what is the build order exactly you are supposed to follow so people can reproduce your stack ?" It's annoying, but for bootstrapping you can break the dependencies. There are lots of ways to to do it. And in the longer term, you can work to break the dependencies; a lot of these cycles are inadvertent.
Fedora/mock situation can be fixed Posted Jun 25, 2013 19:24 UTC (Tue) by nim-nim (subscriber, #34454) [Link]
"The good news is that we've got a lot more cycles to throw at the problem"
The limiting factor is not the build farm cycles. The limiting factor is your deployment process. What good your beefed up build farm that can rebuild the universe in a blink will be, if you can't push the result to users fast enough? Rebuilding for security problems is not here to look good on the builder, to be useful it must be deployed as fast as possible. The more stuff you rebuild to have a perfectly coherent universe the more bits you have to push user-side.
The current systems balance reproducibility with deployment ease.
> It's annoying, but for bootstrapping you can break the dependencies.
And new cycles will be added faster than you can break them. Distros have tried for years to break the cycles with no luck. Developers keep adding them. They don't care about the system costs, they are not system people. An academic from-scratch project could try to eradicate them, but in the real world, Apple (OSX), Google (Android), Sony (PS4) all chose to reuse off-the-shelf software. Restarting from zero is not an realistic scenario nowadays, you get to live with preexisting software, warts included.
Fedora/mock situation can be fixed Posted Jun 25, 2013 20:27 UTC (Tue) by david.a.wheeler (subscriber, #72896) [Link] "The limiting factor is your deployment process. What good your beefed up build farm that can rebuild the universe in a blink will be, if you can't push the result to users fast enough?" - The obvious answer is to rebuild before a major release. For example, before releasing Fedora 32, recompile from the ground up (twice) so that Fedora 32, when it compiles itself, will produce itself. You can do this in a rolling process to avoid making it a "big bang". There are lots of other options too. "new cycles will be added faster than you can break them." - Yes, those are annoying. But they can be handled.
Checking each intermediate step in 'make' Posted Jun 21, 2013 6:49 UTC (Fri) by epa (subscriber, #39769) [Link]
I wonder if 'make' could help by providing a build log. At each stage when a target is built, make hashes individually all the input files and the resulting output. A list of these filenames and hashes becomes a build log. When building again, make can check against an existing log, and stop at the first point where the hashes diverge. This would also be useful in-house just to check that your build hasn't mysteriously changed. (If it has, there might not be a problem, but you should at least *know* about it.)
Checking each intermediate step in 'make' Posted Jun 21, 2013 8:31 UTC (Fri) by oever (subscriber, #987) [Link]
Your suggestion is a good way to find non-reproducible behavior in builds.
I have started working on a tool to do this. It is not a source code patch of make but a runtime patch, i.e. a library that is loaded by LD_PRELOAD. It tracks which files are written and which files are read. It also does this for subprocesses.
The problem is quite complex. The input into a command is manifold: the time, the environment variables, the entire state of the file system.
Here is a short description of what the tool does. Instead of running a command e.g. "make", one runs "env -i LD_PRELOAD=librb-preload.so /usr/bin/make". ("env -i" runs a command in an empty environment.).
At this point you have two options: checking or caching. In checking mode, you run the command again and check if the output is the same. In caching mode, you assume the output will be the same and instead of running the actual command, you retrieve the resulting files from the database.
The very, very rough beginnings of such a tool are here:
Checking each intermediate step in 'make' Posted Jun 21, 2013 9:53 UTC (Fri) by epa (subscriber, #39769) [Link]
Sounds interesting. With C and C++, a lot depends on the header files. Normally, preprocessor expansion is not a separate stage - so if compiling foo.c gives a different foo.o from what you expected, you don't know if it is because the code generation is different or just because a certain #include has changed. To give the most information, makefiles would have to treat preprocessing as a separate intermediate stage, with the preprocessor output written to disk and then read back in for compiling. (The same applies to assembly language before it is converted to machine code, but it seems unlikely that your assembler would ever start making different code.)
Checking each intermediate step in 'make' Posted Jun 21, 2013 14:00 UTC (Fri) by mathstuf (subscriber, #69389) [Link]
This is also a goal of tup. It will actually fail if inputs that weren't specified are read or outputs written. It makes it hard to, say, extract a tarball as a tuprule, so maybe just injecting make (though it should work for ninja as well I expect) is the easier solution rather than trying your buildsystem scripts for tup.
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 9:05 UTC (Fri) by aristedes (guest, #35729) [Link]
This is one of the reasons why, over at the Apache Foundation, we consider that a software release is a release of the *source code*. The binaries are for convenience but aren't really an essential part of the release. They are not what the PMC is tagging as *the release*.
Yes, in practice, most people just grab the binaries, but the distinction between the two is important to keep in mind.
Another important aspect of an project graduating from the incubator and becoming a full blown Apache project is that users must be able to take the source code and compile it themselves. That is, there should be sufficient tools included to make it possible. It is just as important for the tool chain to be released under the open source license as the code itself.
Now, 99% of the time that is really simple: a Makefile, build.gradle or pom.xml. But every so often you collide with a project like OpenOffice which is quite horrendous to build. I am sure most open source foundations have similar goals, but being able to easily build the source is essential to attracting new developers to hack at your code.
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 11:25 UTC (Fri) by pboddie (guest, #50784) [Link] This is one of the reasons why, over at the Apache Foundation, we consider that a software release is a release of the *source code*. Certainly, but verifying the origins of binaries is important not just for trust purposes but, as the article points out, also for assessing licence compliance. There seem to be plenty of vendors who throw binaries over the wall (or over the air) for devices that they make - you don't start off with the sources at all - and these vendors typically need to be pressured for the sources and for those all-important build scripts. But every so often you collide with a project like OpenOffice which is quite horrendous to build. Indeed. I believe LibreOffice has made some progress in this area, however. ;-)
Van Den Oever: Is that really the source code for this software? Posted Jun 21, 2013 12:49 UTC (Fri) by TRS-80 (subscriber, #1804) [Link] Vesta is an SCM (originally from Digital/Compaq) that sucks in the entire build environment into the repository, so you can get the same build every time.
Compiler/Linker options to embed the source code in the output? Posted Jun 21, 2013 17:17 UTC (Fri) by hamjudo (subscriber, #363) [Link] This won't scale well to big projects, but for small projects, I'd love it if there was a compiler option to dump the source code into the .o file, and for the linker to copy the source and related data into the executable or shared library. We would need a way to tell the linker what other source it needs to throw in to make a complete distribution.I'm lazy, and disorganized. This would vastly simplify the logistics of complying with the code distribution requirements of the GPL, if the tool chain did the bulk of the work. More importantly, it would ensure that the right version of the source was always with the executables.
Compiler/Linker options to embed the source code in the output? Posted Jun 21, 2013 18:23 UTC (Fri) by rgmoore (subscriber, #75) [Link] The executable or shared library is probably the wrong level to do this kind of thing. But in practice, very little software is distributed as bare exectuables or libraries; instead, it's packaged into bundles like tarballs, package manager files, Windows installer files, etc. It seems to me that the easiest approach would be to program the package creation script so it automatically bundles the source with the binary. As long as this is done consistently, so people automatically get the source to all the necessary libraries, it should make things much simpler. This won't do anything to deal with downstream distributors who strip out the source code to slim down the packages, but they could do the same thing by stripping unnecessary information out of binary objects, too.
Compiler/Linker options to embed the source code in the output? Posted Jun 21, 2013 19:03 UTC (Fri) by hamjudo (subscriber, #363) [Link] Doing it right in the package creation process has always been an option. And this won't prevent anyone from doing it that right way. This would be a handy way for including the source with software that is normally packaged in radically different ways. For example, the software installed on embedded systems at time of purchase.The cost of flash has dropped tremendously in the last few years. A decade ago, it was cheaper to ship a router with a CDrom, than to add another gigabyte of flash to the product. Source code was usually shipped on a CDrom, and it was almost universally the wrong version. If we made the ELF files fatter, we could get rid of the need for the CDrom, and at the same time, dramatically improve the odds of getting the right version.
Compiler/Linker options to embed the source code in the output? Posted Jun 21, 2013 18:28 UTC (Fri) by raven667 (subscriber, #5198) [Link]
This sounds a lot like FatELF but including the source and build environment descriptions as well. A system built this way could to just-in-time compilation of source to native executable format wherever programs are installed which would be interesting.
Compiler/Linker options to embed the source code in the output? Posted Jun 21, 2013 19:58 UTC (Fri) by viro (subscriber, #7872) [Link]
Preprocessed source is _not_ enough for e.g. GPL compliance, and for a very good reason (seriously, take a look at it someday). And real source would be bloody painful - what would you put into object? something.c + all header.h that had been pulled in?
Compiler/Linker options to embed the source code in the output? Posted Jun 21, 2013 20:41 UTC (Fri) by hamjudo (subscriber, #363) [Link] Pull it all in, and have the linker throw out all but one copy of each header. Once source is preprocessed, it isn't really source anymore. You need the real stuff. You also need all of the command lines, both what was fed to make, and what make eventually used to invoke the compiler. Source without a real set of preprocessor flags isn't complete either.This will take time to percolate out to the world at large. Moore's law and its associated regulations for bandwidths and storage capacity will blunt some of the pain. Although, there is no getting around the problem that some hairballs of .h files are insane. Having the compiler copy the hairballs around would highlight the insanity.
Compiler/Linker options to embed the source code in the output? Posted Jun 23, 2013 22:44 UTC (Sun) by Tobu (subscriber, #24111) [Link] Someone should do a bit of toolchain work and bundle the source and a few git hashes with DWARF debug metadata; -ggdb3 already includes all the macros and source offsets, resolving the offsets to source at link time should be quite workable.
|
|||||||||||||