Not logged in
Log in now
Create an account
Subscribe to LWN
Pencil, Pencil, and Pencil
Dividing the Linux desktop
LWN.net Weekly Edition for June 13, 2013
A report from pgCon 2013
Little things that matter in language design
On one side there are arguments that a change that is made by free software purists that happens to break pre-existing programs is good - because FLASH is one of the broken programs...
On the other side are arguments that users systems are exposed to corruptions due to changes in the behaviour of a library call made for a marginal optimisation of a utility function.
To put it in perspective: I do not want the software I rely on to have ONE randomly inserted bug activated for a 200% improvement of its overall performance.
That bug could be the one a hacker uses to observe my credit-card details when paying for LWN subscription.
The proposed benefit is 20% of 2%, or 0.4%. The possible cost is my bank account.
I hope that the packagers of the distributions do the sensible thing.
That is: pull that change out and shoot it.
This isn't either/or. Phase in such changes instead!
Posted Nov 11, 2010 22:57 UTC (Thu) by dgm (subscriber, #49227)
Posted Nov 11, 2010 23:09 UTC (Thu) by dafid_b (guest, #67424)
"Actually I think we may have first seen this with squashfs. Problems showed up right before the F14 alpha. Phillip found the cause of the problem was using memcpy instead of memmove."
So there are at least two bugs exposed by this change in Glibc.
There may be more. There are vast number of applications out there still waiting to be tested.
It is just impolite to cause users to do the testing when you don't have to.
Posted Nov 12, 2010 4:31 UTC (Fri) by mrshiny (subscriber, #4266)
Posted Nov 17, 2010 15:14 UTC (Wed) by meuh (subscriber, #22042)
It's not a bug. And it doesn't affect all users.
Hopefully, legitimate uses (regarding to specification) of memcpy() are not affected by the optimisation in newer glibc.
Posted Nov 17, 2010 15:52 UTC (Wed) by mrshiny (subscriber, #4266)
Yes, it is a bug. Sure, the application is responsible for using APIs properly. But here we have a situation where a library has worked one way for years, and then suddenly works a different way. There was no way for the apps in question to detect the bugs because the code worked perfectly before. Now, due to a library upgrade, those apps don't work. In some cases there is data corruption. The corruption might happen silently. There is no way to be sure that this change is not quietly damaging untold amounts of data without auditing every use of memcpy everywhere to ensure that it is doing the right thing.
And this means that not only do you have to fix all source code which is wrong and issue new binaries, but you shouldn't upgrade to this version of Glibc because you might have an app somewhere that wasn't fixed, or isn't fixed in the version you have installed.
Glibc is a critical library in the system. Almost every program uses it. As such, it is their responsibility to treat ABI changes very carefully. Sure, this is not a change in the specification, it is an unintended consequence and it's due to those stupid lazy programmers who didn't read the spec or didn't care or whatever. Or inadvertently introduced errors when their code was changed. Or changed something without realizing that this change would result, somewhere, in a call to overlapping memcpy. Given that the bug was hard to identify (at least for some cases), and given that Glibc has symbol versioning, maybe they should use it?
Your last sentence sums up the problem: "Hopefully legitimate uses are not affected". I think we should expect stronger guarantees from glibc than "hopefully".
Posted Nov 17, 2010 16:09 UTC (Wed) by meuh (subscriber, #22042)
Posted Nov 17, 2010 16:23 UTC (Wed) by xilun (subscriber, #50638)
There is also no way to be sure that this change is not _fixing_ untold amounts of data corruption when the memcpy is done backward without auditing every use of memcpy :)
Anyway, C being what it is, this is a little ridiculous to do a fixation on that particular change, because some other changes exposing bugs are done every day, hundred at a time. So really, you have no way after ANY upgrade to be sure that memory corruption won't mysteriously happens when they previously did not. If that's a problem for you, don't ever update anything => problem magically solved.
> given that Glibc has symbol versioning, maybe they should use it?
Nope. Symbol versioning is for ABI changes, and symbol versioning does not even pretend to automatically solve every problem ABI changes has been shown to cause. The memcpy implementation change is not even an ABI change.
> Your last sentence sums up the problem: "Hopefully legitimate uses are not affected". I think we should expect stronger guarantees from glibc than "hopefully".
There was a problem only in the sentence. The "hopefully" is not needed. Legitimate users of memcpy will not be affected.
Posted Nov 12, 2010 0:08 UTC (Fri) by xilun (subscriber, #50638)
I fail to see how my previous post, which you replied to, is in any way related to free software purist happy to break Flash.
Indeed I think you did not even read it.
So I'll make an executive summary (but with new elements, for those who follow): in http://www.coding-guidelines.com/cbook/cbook1_2.pdf ; read, starting at pdf page 183, 3.4 behavior, 3.4.1 implementation-defined behavior, 3.4.3 undefined behavior, and 3.4.4 unspecified behavior. You will then hopefully understand why it would indeed be *dangerous* for security (not even talking about performance) in the long term if a widely used implementation starts giving guarantees defining "undefined behaviors", or if the maintainers of such implementation start acting like there seems to be some guarantees. (Think about other compliant implementations.)
If you don't like implementation-defined, undefined and unspecified behaviors in programming languages, use Java. I'm indeed starting to wonder if Linus does not secretly dream about writing operating systems in Java -- look at: some of his responses during the NULL-page mapping debacle, GCC adding optimizations taking advantages of undefined behaviors on integers, and his position on this memcpy implementation.
> On the other side are arguments that users systems are exposed to corruptions due to changes in the behaviour of a library call made for a marginal optimisation of a utility function.
Users systems are exposed to corruptions because they wrote code having undefined behavior in the first place, and there should be neither surprise nor scandal when code containing faulty constructs having undefined behavior starts to behave in an undefined way, because that's precisely the definition of what "undefined behavior" means.
Undefined behavior could has well change observable behavior depending on your power supply, the phase of the moon, and the fact Linus has been personally annoyed by a random bug (the last cause being the most probable in those examples, which is a little weird from an economical perspective, but oh well). Blame glibc maintainers all that you want, but you'll soon have multiple targets when the next advance in GCC expose other bugs caused by other undefined behaviors.
> To put it in perspective: I do not want the software I rely on to have ONE randomly inserted bug activated for a 200% improvement of its overall performance.
Under which perspective bugs are not "randomly" inserted? (given a non malicious intent in the first place). Would you be OK with "ONE randomly inserted bug activated" because of a change for support of a new hardware, or a new feature. Do you realize that even a bug fix can activate other bugs? Do you realize that you can easily avoid all that kind of trouble by NOT upgrading your system library ever, if you really want to? Do you understand that optimizations made at system level follow a different economic than optimizations made at application level? Do you understand that compiler/library evolution have participated in the moore law, and that your computer would be maybe 4x slower or produce 4x more heat if we still were in the naive compiler era and if low level layers had not been updated to be efficient with modern processor architecture?
> The proposed benefit is 20% of 2%, or 0.4%. The possible cost is my bank account.
Given the nature of the memory corruption, very unlikely to have that kind of security impact (but not 100 % impossible).
What is really funny is that even without the incriminated patch, the memcpy was NOT a memmove (fortunately). This particular Flash call resulted in corrupted data when copying memory in reverse order because the pointers was in a specific order, and the area overlapping. Calling memcpy with the previous GLibc implementation, and probably 99% of implementations existing on earth, will still result in data corruption if memory area are overlapping in the other order.
So I suggest you run your whole system LD_PRELOAD'ing all processes with a library that calls memmove instead of memcpy, if you are worried too much about that.
I also suggest that you immediately start looking for other bugs more susceptible to have big security impacts than this class, and that you also workaround them in weird way instead of fixing them correctly in the first place.
Maybe it would indeed be easier that you take a really old distribution, with a compiler that does very few optimization, and a very simple libc, and stick to it forever. (Well, you'll still have to do the memcpy/memmove replacement trick, but you'll have very few optimizations, so I guess that will make you happy.)
And oh, I forget to tell you: randomly defining "undefined behaviors" without auditing every components involved in both the system and its construction can sometime expose bugs with an high security impact. See the NULL page mapping debacle.
> I hope that the packagers of the distributions do the sensible thing.
> That is: pull that change out and shoot it.
Yeah, they all are reading LWN comments, waiting for your enlightenments.
Posted Nov 12, 2010 1:10 UTC (Fri) by dafid_b (guest, #67424)
However, I think that the conversation we are having in this thread is a bit disjointed because when I say 'user' I mean a 3rd party to this conversation - not the Flash developer, not the Glibc developer, nor the crushfs developer.. but a simple user:). Whereas I think you read my user as 'developer'.
In the later post I liken (the knowing continued) delivery of this change to Glibc to mugging the person (user) who is near (uses the software written by) a jay-walker(developer who used undefined behaviour that used to work in the past).
That does not seem very fair to the user. It is sure to convince most users to stop being Linux users if the change does cause a security issue to happen - and they find out that it was a deliberate choice.
I think a better policy would be to mug the developer (send the crash reports, mocking messages in the trade press, or whatever).
This could be done by putting an intercept layer between Glibc in system tests that any user could load - at a known performance cost - that logs such violations of API requirements.
I would be happy, ecstatic even, to take part in such a mugging, when I am not doing my banking on the system.
Thanks for pointing out my other mistake - I should have said 'randomly activated bugs' rather than 'randomly inserted bugs' - as from both the end-user and developer perspective that is what is happening.
On your other points, I agree. However I think that the problem the points address is developer behaviour, and the person you mug is the user.
end user should not be punished, depending of distro target
Posted Nov 12, 2010 2:28 UTC (Fri) by xilun (subscriber, #50638)
But I would not even be angry against a distribution that makes the choice to not care at all about Flash. I perfectly understand that some can absolutely not care about Flash, in which case an angry user should just do a workaround himself or switch to an other distro, if he indeed is not in the target of the one he used.
Posted Nov 12, 2010 5:03 UTC (Fri) by dafid_b (guest, #67424)
Posted Nov 12, 2010 19:28 UTC (Fri) by charlieb (subscriber, #23340)
And wouldn't it be nice if Fedora 14 were to do this :-)
Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds