Weekly Edition
Return to the Front page
| |||||||||||||
The unstoppable Perl release train?
There are a number of release management styles to be observed in the free
software community, but many of them fall within the two broad categories
of "time-based" or "feature-based". A time-based project fixes a date for
its next release, then adjusts the set of features included in that release
to make the intended date at least somewhat plausible. Feature-based
projects, instead, pick a set of desired features and hold the actual
release "until it's ready." Arguably, the trend over the last decade has
been in the direction of time-based releases. Perl 5 is a relatively
recent convert to the time-based model; the project is now faced with a
decision between making a release with known problems - including possible
security issues - or delaying its 5.16 release.
Contemporary Perl's release schedule is approximately one major release per year. The 5.14 release came out on May 14, 2011, so it is not too soon to be thinking about 5.16. That release is indeed stabilizing with a number of new features. A key aspect of this release is a familiar story: as with most other languages, Perl developers are trying to improve their support for Unicode in all situations. Many developers have participated in this work, but Tom Christiansen has arguably been the most visible. His recent work includes the preparation of an extensive Perl Unicode cookbook demonstrating Perl's Unicode-related features which, he has suggested, are now second to none. As this cookbook was being discussed, Karl Williamson pointed out that some of the examples where Perl is put into full-time UTF8 mode (a very useful mode for contemporary programs) are unsafe because the language does not properly handle malformed strings. At best, such problems could lead to the corrupted strings seen in so many settings where character encodings are not properly handled. But, as Christian Hansen suggested, things can be worse than that:
I would love for this to happen, I have advocated this on #p5p
several times, but there is always the battle of "backwards
compatibility disease". About 10 months ago I reported a security
issue reading the relaxed UTF-8 implementation (still undisclosed
and still exploitable) on the perl security mailing list.
What followed was a classic missive from Tom trying to get to the bottom of the problem. He listed nine different ways to tell Perl to operate with UTF8 and asked how many of them were truly vulnerable to undetected encoding problems. If Perl's UTF8 handling is unsafe by default, he said, it needs to be fixed:
If there's something so important that it must be done everytime to
ensure correct behavior, then that is too important to be left up
to the programmer to forget to do. It needs to be done for him.
And, he said, this situation really needs to be treated as a blocker for the 5.16 release; to do otherwise would be to delay the fix excessively and cause the world to be filled with bad workaround code. In many projects, the prospect of open security-related problems would at least cause people to think about delaying a release. On the Perl list, though, Tom found little support. Aristotle Pagaltzis responded:
5.16 *must* be released whatever the state of this issue. To not do
so is to fall into the thinking and behavioural pattern that
stifled the release of 5.10 by several years. Perl 5 switched to a
timeboxed release cycle because “this one more thing has to be
polished before we can ship it” meant it never shipped at all.
Ricardo Signes added:
It's definitely true, though, that 5.even.0 releases *are no longer
milestones.* Or, rather, they are milestones in a much more literal
sense than is often meant. 5.16.0 means that we've come about one
year since 5.15.0. It does *not* mean that we have met a series of
goals named at 5.15.0, for example.
And, with those words, the public email discussion faded away. At this point, there is little clarity on which Unicode features are safe to use, what might be required to fix the rest, how many of those fixes might be ready in time for the 5.16 release, or whether programs using UTF8 in Perl 5.16 (and earlier releases) suffer from known-exploitable security problems. Tom, who probably understands these issues better than just about anybody else, said:
Right now I'm very hazy on the real status of all this stuff, and I
am very uncomfortable with the idea of relentlessly charging ahead
toward a release like a freight train with no brakes.
The response he got claimed another train would be coming along in a year and the fixes could catch a ride on that one. Releasing software with known bugs is a common practice; even a project like Debian cannot make the claim that there are no known problems with its releases. To do otherwise would make software releases into rare events indeed. It is also true that time-based releases have value; users know that they will get useful new code in a bounded period of time. Anybody who has watched release dates slip indefinitely knows how frustrating that can be for both users and developers; the 2.4.0 or 2.6.0 kernel releases are a classic example - as is Perl 5.10. So the Perl developers who say that the show must go on are doing so in accordance with many years of free software development experience. That said, releasing software with known, security-related issues in something as fundamental as UTF8 support risks tarnishing the image of the project for a long time. There is not enough information publicly available to say whether Perl's UTF8 problems are severe enough to incur this risk. But it is probably safe to assume that, as a result of this conversation, crackers are looking at Perl's UTF8 handling rather more closely than they were before. Perl may have had some of these problems for years without massive ill effect, but they may not remain undiscovered and undisclosed for much longer. If the problem is real and exploitable, people are going to figure it out and take advantage of it. One would assume that, behind the public positioning, the relevant developers understand what's at stake and are taking the time to understand the scope of the problem. They may not want to stop the release train, but seeing it derailed by a known security problem after release would not be much fun either. A release delay now may prove less painful than a security-update fire drill later. (Log in to post comments)
The unstoppable Perl release train? Posted Mar 1, 2012 7:08 UTC (Thu) by Los__D (guest, #15263) [Link]
If they want to be time-based (without brakes), the only sane thing to do, would be to back out all the UTF-8 changes,though I would guess that they are such a big change that it would be pretty har to do without breaking almost every other patch for the release.
The unstoppable Perl release train? Posted Mar 1, 2012 7:40 UTC (Thu) by autarch (subscriber, #22025) [Link]
The first support for UTF-8 was added to Perl in 5.6.0, release back around 2000 or so. I'm pretty sure that of the sane things to do, attempting to back out 12 years worth of changes is not one of them.
The unstoppable Perl release train? Posted Mar 1, 2012 8:04 UTC (Thu) by Los__D (guest, #15263) [Link]
Heh, indeed.
I assumed that the security issue was with a new way of handling UTF-8. If the boilerplate code needed for security is nothing new, then I agree that the release should go ahead, as it is already out there.
Encoding-related vulnerabilities Posted Mar 1, 2012 12:36 UTC (Thu) by tialaramex (subscriber, #21167) [Link]
I have no insider knowledge of Perl (my Perl is about as good as my French, I can make myself understood but there's no mistaking me for a native) and no insider knowledge of this specific security vulnerability
BUT, I do know about UTF-8 and the usual routes for bad handling of text encodings to open up vulnerabilities are stuff like:
1. Mistakenly accepting non-canonical encodings as an end run around character restrictions, e.g. the forward slash '/' in Unix pathnames is U+002F, correctly expressed in UTF-8 as a single byte 0x2F the same as ASCII. But the UTF-8 byte sequence 0xC0 0xAF (if I did my math right) also decodes as U+002F. This "over long sequence" is prohibited by the UTF-8 standards, but a cheap-and-cheerful decoder might allow it anyway. The bytes 0xC0 and 0xAF don't match 0x2F so it's possible you can sneak a leading forward slash past a routine intended to filter them out.
2. Error-recovery routines that mangle strings in ways useful to an attacker. The Unicode specifications explain that if you can't or won't report an encoding error (e.g. return error code, throw exception) when processing you should inject the reserved codepoint U+FFFD "replacement character" in place of each code unit you couldn't handle. But it's much /simpler/ to just discard the erroneous bytes. Doing that will allow attackers to re-construct a string that would have been blacklisted earlier in processing (e.g. sneaking the word "escape" past a filter by injecting a nonsense byte between "esc" and "ape").
3. Dumb operations leading to surprising errors. A sixteen byte ASCII string can always be safely divided into two eight byte ASCII strings, the resulting strings may not make sense, but they're legal. You never get an error from trying to do this. The same is not true in UTF-8, strings may only be correctly cut at the boundary between codepoints, dividing them otherwise may result in illegal strings. You may get an unexpected error or the strings may not be eight bytes long as expected. Code that's handled everything thrown at in ASCII suddenly "dies" when attackers send gibberish that's alleged to be UTF-8 instead.
Of course Perl is always innovating so it's possible they've genuinely found a new and exciting way to screw up, but it's more likely to be something like the above.
Encoding-related vulnerabilities Posted Mar 1, 2012 17:38 UTC (Thu) by erwbgy (subscriber, #4104) [Link] Thank you. Informative comments like this are what makes reading LWN such a pleasure.
The unfixed Perl security process? Posted Mar 1, 2012 15:19 UTC (Thu) by louie (subscriber, #3285) [Link]
Seems like that is the right title for this post. I'm a big fan of unstoppable release trains, but it presupposes that the rest of your release process is not screwed up. In particular, it assumes that critical bugs are identified early in the process and released. Sounds like that is badly broken here.
The unfixed Perl security process? Posted Mar 2, 2012 6:15 UTC (Fri) by speedster1 (subscriber, #8143) [Link]
> In particular, it assumes that critical bugs are identified early in the
> process and released. Sounds like that is badly broken here.
That sounds ideal, but what sort of release process could reliably accomplish that goal with respect to security bugs? Tell people to have their security-related discussions during the first half of a new release, analogous to the kernel merge window?
The unfixed Perl security process? Posted Mar 6, 2012 16:57 UTC (Tue) by man_ls (subscriber, #15091) [Link] I am assuming give a high priority to fixing critical issues before adding new features. Apparently this particular problem has been known for a long time but has gone unfixed.
The unstoppable Perl release train? Posted Mar 1, 2012 15:55 UTC (Thu) by autarch (subscriber, #22025) [Link]
This article isn't very good, and I'm disappointed at LWN for publishing it.
The problem is that it conflates two separate issues.
Tom Christiansen reported some bugs in Perl's Unicode handling. Bugs are bad, and should obviously be fixed. Perl is on a fixed release schedule of yearly major releases, and currently is in a freeze leading up to the next major release (5.16.0), planned for April of this year. This means that most bug fixes will not make it in for 5.16.0. Fortunately, 5.18.0 is only a year away.
In addition, fixes checked in for 5.18.0 will be considered for inclusion in 5.16.1 (or .2, .3, etc) if they are backwards-compatible.
But I'll also point out that while the bugs have been reported, no one has actually stepped up and fix them. Discussion of delaying the 5.16.0 release for these fixes is especially ridiculous in that context.
Separately, there is also a possible security issue with Perl's Unicode handling, reported to the Perl security list 10 months ago. None of us discussing this here know what that is, because it has not yet been disclosed (and according to Christian Hansen, it's not fixed yet).
Given that it was reported 10 months ago, we can say for sure that is is present in the existing stable release (5.14.2). If I had to guess, I'd guess that it's also present in the 5.12 series of releaess, and probably in 5.10.1 and 5.8.9, both of which are still widely used in the wild.
Presumably if this was easy to fix then a fix would already have been released, so let's assume that it's hard to fix. The security bug will be fixed when it is fixed. But to release it, I suspect the Perl developers will want to coordinate a new release of at least 5.14 and probably 5.12. They may also want to coordinate with distributors to work on patches for 5.8 and 5.10, since those are both still being supported by various distros.
Whether or not 5.16.0 has been released in the time between now and the security fix is pretty much irrelevant. If 5.16.0 has been shipped, then the security fix will necessitate a 5.16.1 release, but so what?
Releasing 5.16.0 will hardly make using Perl *less* secure, and delaying the 5.16.0 release will not make Perl *more* secure.
Releasing 5.16.0 will hardly make Perl *less* buggy, and delaying the 5.16.0 release will not make Perl *more* buggy.
So what exactly is broken about the Perl release process here?
The unstoppable Perl release train? Posted Mar 1, 2012 15:57 UTC (Thu) by autarch (subscriber, #22025) [Link]
Doh, got my more and less reversed in the second use. It should say ...
Releasing 5.16.0 will hardly make Perl *more* buggy, and delaying the 5.16.0 release will not make Perl *less* buggy.
The unstoppable Perl release train? Posted Mar 1, 2012 18:11 UTC (Thu) by felixfix (subscriber, #242) [Link]
I would say that your summary is what is broken here.
By your own admission, none of us mere mortals know what the Unicode bug is or how it affects Perl. You then pretend that you do know how serious it is and that should in turn dictate not pausing the release schedule.
You can't have it both ways. Me, I tend to trust Tom Christiansen and the other Perl gurus. If Tom thinks the release should be delayed, that carries far more weight then you saying it shouldn't. If other Perl gurus think Tom is wrong, that also carries far more weight than your pronouncement. I have faith they will work things out.
To call this article "not very good" is pretty arrogant coming from someone who thinks he has more knowledge than the perl gurus.
The unstoppable Perl release train? Posted Mar 1, 2012 18:26 UTC (Thu) by autarch (subscriber, #22025) [Link]
You didn't read what I wrote very carefully, nor do you know who I am.
First, I am one of the Perl gurus, though Tom C can clearly claim higher level guru status than me (on the official Perl Guru rating system I'm a 3 to the power of 𝛑, but Tom is at least a 7 to the power of 3𝛑). But hey, I did release Perl 5.15.6 (https://metacpan.org/release/DROLSKY/perl-5.15.6/), and I've worked on the Perl core a bit (mostly docs).
Second, the "Perl gurus" obviously don't all agree on what the current problem is. Tom C says one thing. Many others disagree with his assessment of the urgency of these bugs, which the article does make clear (Aristotle and RJBS are both also Perl Gurus, though I haven't calculated their exact ratings yet).
But here's the real point ...
These bugs that Tom brought up have existed for quite a while. Note that they are not all security bugs. The security bug mentioned in the article has not been disclosed, and may be something not in Tom's list. The security bug has *also* existed for a while (since 5.14 at the very least).
If we *don't* release Perl 5.16.0 in April, then the latest stable release (5.14.2) will have all of these bugs.
If we *do* release Perl 5.16.0 in April, then the latest stable release (5.16.0) will have all of these bugs.
This is why delaying the release makes no difference. No matter what we do, the latest stable release will have all of these bugs, including the unknown security bug!
The only reason to delay the release is to fix *release-blocking* bugs. These are defined as bugs which introduce *new* regressions since the last stable release. If Perl 5.14.2 didn't have these Unicode bugs, then these would probably be considered blockers, but that's not the case.
The article totally misses this point, which is one reason why I don't think the article is very good.
The unstoppable Perl release train? Posted Mar 1, 2012 22:24 UTC (Thu) by gerdesj (subscriber, #5446) [Link]
Willy waving about relative guru powers is silly. Do you have any idea what the difference between 3^PI and 7^(3*PI) actually is? It makes you look daft.
Reporting on unspecified security problems in a core part of many systems seems to me a really good point to the article. I'll take an informed article with references from the grumpy ed over a comment from a participant any day. Especially an article that provides references to both sides of the argument and invites me to make my own mind up.
Cheers
The unstoppable Perl release train? Posted Mar 2, 2012 1:04 UTC (Fri) by autarch (subscriber, #22025) [Link]
I was responding to someone who claimed I was unqualified to declare that the article wasn't very good.
The bit about guru levels is of course quite silly. I was joking!
The unstoppable Perl release train? Posted Mar 2, 2012 1:33 UTC (Fri) by felixfix (subscriber, #242) [Link]
My main point was no more about how qualified you are than your main point was about how qualified you are.
My main point was that you first admit you don't know how important the security issue is, then claim that the security issue is not important enough to delay the release.
You can't have it both ways.
The unstoppable Perl release train? Posted Mar 2, 2012 15:04 UTC (Fri) by autarch (subscriber, #22025) [Link]
Delaying the release won't get this security issue fixed any faster.
Also, Tom Christiansen was not suggesting that the release be delayed to fix this particular issue. He wanted it to be delayed to fix the bugs he was reporting. He is not the person who reported the security issue.
The unstoppable Perl release train? Posted Mar 1, 2012 23:43 UTC (Thu) by jhhaller (subscriber, #56103) [Link]
Not having any bone to pick on either side, it appears to this outsider that fixing security issues are not as big of a concern than maintaining a consistent release schedule. Holding back a release because of a vulnerability gives everyone the impression that security is important and a project priority. Releasing it says we will get to it when we get to it, or that Unicode isn't thought to be important. Even if a statement was made that we are working on the bug, the release will go out shortly. but that a patch will be released shortly would be a better than only saying the release schedule is sacrosanct.
This article helped shine a light on this issue, which outsiders not watching Perl mailing lists would never have seen otherwise. If this puts enough pressure to get the bug fixed in a timely fashion, then the article served it's purpose.
The unstoppable Perl release train? Posted Mar 2, 2012 1:08 UTC (Fri) by autarch (subscriber, #22025) [Link]
Holding back a release for a security issue would make sense if the security issue were not already present in the last stable release, I agree.
I'm not sure of the exact details of the security issue, but given that it was first reported 10 months ago (and was probably reported against the stable release at that time), we can assume that existing stable versions of Perl are affected.
Releasing another stable release of Perl which is affected by the same problem really doesn't make a difference, security-wise.
The unstoppable Perl release train? Posted Mar 2, 2012 4:04 UTC (Fri) by cmccabe (guest, #60281) [Link]
Common sense would suggest fixing the gaping security hole before making another release. Luckily we live in an enlightened age when common sense has been replaced by "process" (formerly called "bureaucracy").
The unstoppable Perl release train? Posted Mar 2, 2012 15:06 UTC (Fri) by autarch (subscriber, #22025) [Link]
Your assertion only makes sense if you think that making the next release and fixing the security hole are mutually exclusive. They're not.
The unstoppable Perl release train? Posted Mar 2, 2012 14:40 UTC (Fri) by xdg (guest, #83285) [Link] autarch is correct that the security issue is present in all stable releases of Perl since Unicode support was added. There are no reported exploits. Programs that follow the Perl 5 Security guidelines and the Security Implications of Unicode guidelines are unlikely to be affected. When the issue is addressed, it will be backported to all supported Perl 5 stable release series, per the Perl 5 Support Policy. The release schedule of Perl is irrelevant. Holding up the Perl 5.16 release wouldn't get the issue fixed any faster and would merely hold up the release of other bugs fixed in the Perl 5.15 development cycle.
The unstoppable Perl release train? Posted Mar 2, 2012 20:37 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]
> autarch is correct that the security issue is present in all stable releases of Perl since Unicode support was added.
What is the bug? that's information that I haven't yet seen in this discussion.
The unstoppable Perl release train? Posted Mar 4, 2012 5:27 UTC (Sun) by jmayer (subscriber, #595) [Link]
My reading of the article is different than what you "Perl guys" are reading into it: With the new release there will be "complete" Unicode support, it will be the "if you haven't used unicode before, do it now release". So if there are security problems in the unicode handling in Perl and more people start using the unicode features these problems will be in more and more programs. How many people who write Perl scripts have actually read the security guidelines - probably well below 50%. Many people I know learn mostly from examples, not from manpages.
The unstoppable Perl release train? Posted Mar 4, 2012 19:39 UTC (Sun) by xdg (guest, #83285) [Link] If you're reading this as "the Unicode release", then the author has (probably unintentionally) misled you. Unicode itself is a moving target and Perl has continued to make significant stride to improve how it handle Unicode semantics in the last couple releases. See Unicode Overhaul from the 5.12 release notes and Unicode in the 5.14 release notes. Perl 5.16 continues with this trend of incremental improvements. As for how many people read the security-relevant sections of manpages, that's an issue for any language or tool. Most tools can be used insecurely, dynamic languages particularly so. I would hope that anyone writing or deploying code where security does matter would read relevant manpages.
|
|||||||||||||