LWN.net Logo

On stability

On stability

Posted Sep 9, 2010 4:47 UTC (Thu) by roelofs (guest, #2599)
In reply to: On stability by ringerc
Parent article: Debian squeezes out Chromium

The big issue with fast-changing apps and stability is that they make it impossible to win the "bug race".

Indeed. But the bigger issue, I think, is: what makes Firefox and Chromium so special that they have to pop out a major new, compatibility-busting release every few months? Internet Exploder certainly doesn't; Safari doesn't (AFAIK); and neither does Opera (also AFAIK). This seems like a classic case of cranio-rectal impaction on the part of upstream.

(Of course, if there is a reasonable excuse, I'd love to hear it.)

Greg


(Log in to post comments)

On stability

Posted Sep 9, 2010 5:32 UTC (Thu) by drag (subscriber, #31333) [Link]

> Indeed. But the bigger issue, I think, is: what makes Firefox and Chromium so special that they have to pop out a major new, compatibility-busting release every few months?

It's probably related to the issue of how Linux distributions can't give enough of a crap to put serious effort to care about binary compatibility with applications. You know; so it's possible for software vendors to provide and users to take advantage of newer features in newer software and/or rely on proven older software without tearing their hair out.

It's probably a lot easier for Mozilla and Google to simply bundle the dependencies they require with their software then it is leave it up to the distributions and package management systems to do it for them.

On stability

Posted Sep 9, 2010 6:32 UTC (Thu) by rvfh (subscriber, #31018) [Link]

I think Greg's question was not about bundled libraries, but about the rhythm at which new versions come out for Firefox/Chromium compared to, say, Internet Explorer.

IE v8 has been out for some time now, and it looks like there two schools of thought inn the browser landscape: the fast and furious on one side and the stable on the other.

It would be interesting to understand why Firefox is still a so fast moving target after so many years of existence.

Well, yeah.

Posted Sep 9, 2010 6:56 UTC (Thu) by khim (subscriber, #9252) [Link]

It would be interesting to understand why Firefox is still a so fast moving target after so many years of existence.

Because the web is changing? 3D, Drag-N-Drop, WebStorage, WebM, file access, etc - all these features can only be used by web developers if old versions without them are declared obsolete. Microsoft's answer to this challenge is called Silverlight: you can use all these nifty features there already and then it does not matter if the user uses IE6 or IE9. Chrome/Firefox are trying to propose different answer, but it'll only work if features will be pushed to the end users. It does not matter for the web developer if the feature is committed to git respository year ago or just yesterday, what does matter is when the feature is pushed to end users. So it does not matter when new version is released, what does matter is when old version is killed: Google certainly showed is nicely with Chrome and Firefox followed.

Well, yeah.

Posted Sep 10, 2010 0:38 UTC (Fri) by ras (subscriber, #33059) [Link]

I don't think it is the web. Yes, HTML 5 is pushing things along a bit now. But the last major revision was HTML 4, which was released in 1998. That is hardly a frantic rate of change. Ditto for the other major underlying standards - CSS, SVG, Javascript. WebM has nothing to do with it - it is just a codec, a plugin.

I can think of two reasons for pace browsers are developed at. The first is because it is a monumental task, and when you are undertaking a monumental task break it down in to small bits and release early, release often, otherwise you will be overwhelmed. This is because, as standards go, those published by W3 are real pricks. We have enormous teams of programmers implementing them. As I said they haven't changed much in 12 years, yet a seemingly simple thing like rendering ACID3 correctly is a huge challenge. This is what happens when you publish the standard before writing the code. The IETF's policy of producing working code then publishing the standard is how it should be done. The difficulty in implementing a fully compliant HTML, CSS, and SVG are great examples of why it should be done that way.

The second reason is to do with marketing. I used to use, and still occasionally use Firefox 1.0.1. It has UI bugs, it crashes, and it renders things badly. But it was fast, and if they had just fixed those bugs I would be happy running it today. (Although maybe not for much longer, given the advent of HTML 5.) Fixing those bugs didn't require millions of lines of code to change every few months. What does require a new release every few months is a mindshare competition. (Why do you think Ubuntu does it?) What triggers most of those millions of lines of changes is new eye candy - something like rearranging the tabs and title bar, or the introduction of an "awesome bar".

So it is a combination of those two things - the size of the task means implementing a web browser will take 100's if not 1000's of man years, and that means a steady stream of releases as you do it. This is not unlike we see with the kernel or any other large project release. We know Debian can handle that. But them you mix those necessary changes with avalanche of bubble and froth created by a mind share competition - ie change for no reason than it generates publicity, and you become incompatible with a distribution that values stability over most over things.

Well, yeah.

Posted Sep 10, 2010 1:49 UTC (Fri) by iabervon (subscriber, #722) [Link]

Actually, the practical effect of the standards being such a pain is that, despite there not being new blessed versions of the standards documents, there is a lot of change in the de facto standards. This includes things that have always been in the standard, but which nobody previously expected to work. It also includes things that were never specified, but where enough popular browsers did something sensible and the same that people came to rely on that being available. Just because nobody wrote up standards documents of what was expected for a while doesn't mean that the de facto standards didn't keep changing.

Well, yeah.

Posted Sep 10, 2010 2:23 UTC (Fri) by ras (subscriber, #33059) [Link]

> there is a lot of change in the de facto standards.

I disagree. The de facto standards trail the real ones by a large margin.

That is because the de facto standards are to a large extent determined by the oldest browser in use. Until very recently IE 6, which was released in 2001, and it didn't comply with the existing standards in force back then. Google and Facebook only just stopped supporting it.

This "the browsers must change because the web is changing" thing is a complete furphy. Certainly the web is changing. But that is because people are writing millions of millions of lines of javascript as they figure out how to use this "new" platform. But just because the stuff we build on top of the underlying platform changing at a dizzying rate doesn't mean the platform itself must be changing. On the contrary, it hasn't, until now with HTML5. Inventing new ways to do web pages no more depends on HTML/javascript change than the advancement of the linux kernel depends on gcc changing.

Well, yeah.

Posted Sep 10, 2010 20:57 UTC (Fri) by njs (guest, #40338) [Link]

Sure, but one result of those millions of lines of javascript is that developers are running into limitations of the web platform; the platform has to change if it wants to be competitive with Flash/Silverlight/etc.

On stability

Posted Sep 9, 2010 6:54 UTC (Thu) by josh (subscriber, #17465) [Link]

>> Indeed. But the bigger issue, I think, is: what makes Firefox and Chromium so special that they have to pop out a major new, compatibility-busting release every few months?

>It's probably related to the issue of how Linux distributions can't give enough of a crap to put serious effort to care about binary compatibility with applications. You know; so it's possible for software vendors to provide and users to take advantage of newer features in newer software and/or rely on proven older software without tearing their hair out.

You have that entirely backward. Linux distributions are the *only* ones that end up actually addressing library compatibility issues. Software vendors used to systems with no sensible library system (Windows and OS X) treat Linux exactly the same way, either because they want to have a pile of forked upstream libraries without pushing those changes upstream or maintaining a real library fork, or simply because they don't care about handling Linux correctly and the least common denominator requires shipping everything with the application.

Meanwhile, Linux distributions get stuck trying to figure out whether they can build each crazy upstream application against the system version of the library without breaking whatever crazy assumptions the bundled version allowed, and without rewriting large parts of the application's build system.

On stability

Posted Sep 9, 2010 9:06 UTC (Thu) by fb (subscriber, #53265) [Link]

> or simply because they don't care about handling Linux correctly and the least common denominator requires shipping everything with the application.

There is also the fragmentation that makes handling things correctly harder. From the perspective of the software vendor "Desktop Linux" is a blip in the radar, and a very fragmented one. (How many Fedora, Ubuntu, Suse, Debian etc versions are there to test against?)

The software vendor can rationally decide that there will be less users hit by a bug if they put even more resources testing it on Windows and MAC/OS than trying to test things on this large matrix of different Linux distributions and all their versions in use.

On stability

Posted Sep 9, 2010 17:05 UTC (Thu) by josh (subscriber, #17465) [Link]

That's what "unstable" versions of distributions are for; users of those distributions will happily do the testing for you, and report a pile of bugs either upstream or by way of the distribution. That still doesn't explain shipping the (modified) source code to umpteen libraries in the application source tree.

"Handling things correctly" doesn't mean "test with every obscure Linux distribution", or even "test with the top N distributions". It means "make sure it builds in whatever reasonably up-to-date distribution the developers run, handle shared library versioning sanely, document the list of build dependencies, let Linux distributions package it, and work with them when they report bugs to you".

Well, it's a race...

Posted Sep 9, 2010 6:46 UTC (Thu) by khim (subscriber, #9252) [Link]

Internet Exploder certainly doesn't; Safari doesn't (AFAIK); and neither does Opera (also AFAIK).

Internet Explorer certainly released often enough back when it had competition (IE1: August 1995, IE2: November 1995, IE3: August 1996, IE4: September 1997, IE5: March 1999, IE55: July 2000, IE6: August 2001). When it reached 90%+ market share Microsoft decided to kill the HTML and replace with with XAML. When that failed it started released new versions again - albeit still sluggishly. Safari enjoys MacOS lock-in (Windows version is a joke) while browser which don't have such lock-in support need to innovate faster, so both Firefox and Opera released few versions per year (sure, they were named "minor versions", where Chrome names it's versions "major", but this is just PR). For example Opera released version 10.60 just two months ago.

The only recent difference is faster obsolescence of older versions - and this is related to HTML5 push: if we are proposing it as a viable alternative to Flash or Silverlight then we must guarantee that new features will be accessible to majority of users quite fast. And the only way to achieve it is to aggressively upgrade clients - like Flash or Silverlight are doing.

Well, it's a race...

Posted Sep 9, 2010 15:53 UTC (Thu) by smoogen (subscriber, #97) [Link]

History of things:

Microsoft Explorer was originally Spyglass Enhanced Mosaic. Spyglass was a company whose idea was that by making browsers companies could specialize then it could get into the corporate market versus the consumer market that Netscape was focused on. Problem is that consumer markets move very very quickly and corporate ones do not. So Spyglass ended up in a game of catchup with its 5 programmers against Netscapes 20. However, Spyglass thought it had an ace in the sleeve with a deal with Microsoft.. until they realized that Microsoft could hire hundreds of engineers to rework their source code.

IE1->IE4 were mostly Spyglass code with lots of additions from Microsoft. IE5 was mostly Microsoft with some stuff from the remains of Spyglass (who had gone away in 1998 or so because they had not asked for a per copy payment from Microsoft in exchange for the source code... ) I have been told that sometime after Microsoft reached 90%, they repurposed most of the engineers working on the browser to other projects thus the slow down of 'features' until Mozilla restarted the race.

On stability

Posted Sep 9, 2010 8:46 UTC (Thu) by epa (subscriber, #39769) [Link]

What do you mean by 'compatibility-busting'? New Firefox releases (and, I assume, Chrome and Chromium) have pretty good compatibility with everything except old plugins. A new release may add features, but the web being what it is, it's rare for browsers to break compatibility with existing sites or web applications.

That is why most distributions, including the commercial ones with long support windows, are happy to package the latest browser versions as they come out. Keeping your users on the same four-year-old browser (with only security fixes backported) is not long-term support worthy of the name.

On stability

Posted Sep 9, 2010 9:29 UTC (Thu) by NAR (subscriber, #1313) [Link]

A new release may add features, but the web being what it is, it's rare for browsers to break compatibility with existing sites or web applications.

Actually Opera 10.60 sometimes crashes on gmail, so I went back to 10.10. Bug report was sent.

On stability

Posted Sep 10, 2010 23:43 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

A new release may add features, but the web being what it is, it's rare for browsers to break compatibility with existing sites or web applications.
Actually Opera 10.60 sometimes crashes on gmail, so I went back to 10.10. Bug report was sent.

I assume accidental incompatibility, which this surely is, isn't what the OP had in mind when complaining of compatibility-busting updates.

One thing I thought of when I read that is compatibility with user procedures. If the user knows how to use Opera 59, will he be able to use Opera 60 the same way? In the eight years or so I've been using Opera, both on Linux and Windows, I've upgraded two or three times and each time features I used disappeared. Sometimes they disappeared for good; other times they went into hiding, bound to a different key or something. Consequently, my policy now is not to upgrade. If I were using a system where updates were essentially mandatory, I'd be pretty bothered.

On stability

Posted Sep 9, 2010 17:04 UTC (Thu) by foom (subscriber, #14868) [Link]

New firefox requires new xulrunner and rendering libraries, which are not always compatible, and may require recompiling all dependent applications. If it was only firefox and thunderbird, and nothing else used the rendering engine, then sure, you could simply upgrade to the latest version without thinking. But there's other *desktop* apps involved too...

On stability

Posted Sep 9, 2010 21:41 UTC (Thu) by roelofs (guest, #2599) [Link]

What do you mean by 'compatibility-busting'?

I meant more or less what the article was talking about, at least in part--i.e., the bundling of custom, system-incompatible libraries/toolkits such as Chromium and Webkit. (Other articles have covered Firefox's bundling and occasional forking of system libraries, not to mention its API disaster called "xulrunner.") But beyond that, there's the issue of "self-compatibility," which is what's relevant to the backporting of security fixes. Granted, it's unusual to ding a project on the pace of changes to its internals, but then again, in today's desktop systems there's no greater attack surface than the web browser (and its dependencies). Firefox's never-ending stream of vulnerabilities makes 1990s sendmail look good.

I'm not a complete luddite; I get the need for apps that are as central to the user experience as browsers are to innovate, add features, etc. But with that great power comes great responsibility--i.e., to make the browser significantly more secure than the average desktop app--and I'm not seeing an acknowledgment of that responsibility. Indeed, the short support cycles and general level of code churn that limits the ability of others to provide such support are arguably an abdication of that responsibility. (And, for what it's worth, I really don't see a need for the feature cycles to be so rapid. What are the appalling omissions in, say, a 2007 browser--or even a 2009 one--that are blocking the deployment of critical new web stuff?)

Note that nothing I've said implies that distributions should not be able to package newer releases if that's what makes sense for them. My beef is with the other end, i.e., development practices that penalize those distros (or end users) that don't want to upgrade more than once every couple of years.

Greg

On stability

Posted Sep 9, 2010 22:52 UTC (Thu) by jspaleta (subscriber, #50639) [Link]

look at it this way. Google has a master plan for its Chrome browser that quite frankly does not include the needs of other linux distributions or their users. Chromium the project has a codebase which is the leading edge of that plan for Chrome the product.

Do you really think that ChromeOS is going to have a browser with this sort of rate of churn? The rapid rate of development for Chromium right now has a lot to do with Google's overall plan for its own product line. They have real consumer device targets in mind for Chrome and ChromeOS and what you are seeing is the development run-up towards very specific end-goals.

Lets face it, other linux distributions are not the target audience and are not driving the development curve. But the development curve does make sense if you look at stable ChromeOS deployments as the end goal for the rapid development push that is going on now with Chromium. We can get as mad as we want about the reality of that, and its not going to make any difference at all.

-jef

On stability

Posted Sep 10, 2010 15:24 UTC (Fri) by nevyn (subscriber, #33129) [Link]

> Do you really think that ChromeOS is going to have a browser with this
> sort of rate of churn?

Yes. Look at the Nexus One, that has had ~4 download and reboot OS upgrades in the 6 months I've had it.

Release engineering and stability is something old people talk about.

On stability

Posted Sep 10, 2010 16:57 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

Which version of Android are you running? 2.1 as originally shipped or did you upgrade to 2.2?

On stability

Posted Sep 10, 2010 21:03 UTC (Fri) by nevyn (subscriber, #33129) [Link]

2.2 now, I guess. I bought it in Feb. and hit the update button whenever it told me to (which, as I said, is like 4 times).

The apps. on it are even worse, it was a significant timesaver when the last OS update now allowed me to turn automatic updates on for them.

On stability

Posted Sep 10, 2010 18:31 UTC (Fri) by bronson (subscriber, #4806) [Link]

> Release engineering and stability is something old people talk about.

True! I nominate this for a quote of the week.

On stability

Posted Sep 9, 2010 8:51 UTC (Thu) by fb (subscriber, #53265) [Link]

> what makes Firefox and Chromium so special that they have to pop out a major new, compatibility-busting release every few months? Internet Exploder certainly doesn't; Safari doesn't (AFAIK); and neither does Opera (also AFAIK). This seems like a classic case of cranio-rectal impaction on the part of upstream.

When it's the Linux kernel release cycle, the words "release early, release often" come to everyone's mind, and the release speed is something of a lesson to be learned. An achievement of excellence with regards to software development.

Isn't the same thing that Firefox and Chromium are doing? Releasing early, and releasing often?

[...]

Firefox and chromium development cycles are focused on desktop users.

The position of "stability" in the priorities of many Linux distributions are IMO based on a "server room mentality": stability cannot be at _any_ risk, and there is a sys-admin. If new features are needed, the admin will manually do something about it.

Desktop users, in general, live a different life: (i) there is no "professional full-time admin" for the box (it must "just work") (ii) the whole point of that computer is to browse the internet, print flight tickets, and use VoIP. They need the features, and _some_ stability risk is an acceptable trade off. It just so happens that most desktop Linux users are comfortable as sys-admins.

Off-topic: FWIW _all_ desktop Linux users I knew during my PhD who were not comfortable as sys-admins are now MAC users. Every time I asked for the reason to migrate (these were people who had Linux installed at home) the answer was (in essence) that MAC/OS didn't require them to play sys-admin in order to get things to work.

On stability

Posted Sep 9, 2010 10:37 UTC (Thu) by nicooo (guest, #69134) [Link]

> When it's the Linux kernel release cycle, the words "release early, release often" come to everyone's mind, and the release speed is something of a lesson to be learned. An achievement of excellence with regards to software development.

The kernel doesn't bundle 17 different libraries.

On stability

Posted Sep 9, 2010 12:14 UTC (Thu) by fb (subscriber, #53265) [Link]

>> When it's the Linux kernel release cycle, the words "release early, release often" come to everyone's mind, and the release speed is something of a lesson to be learned. An achievement of excellence with regards to software development.
> The kernel doesn't bundle 17 different libraries.

How is that related?

As far as I had understood both "roelofs" and "ringerc" were complaining about release speed, the time a release is maintained, and new features bringing new bugs. AFAICT that has nothing to do with bundling of libraries.

On stability

Posted Sep 9, 2010 13:29 UTC (Thu) by nicooo (guest, #69134) [Link]

The excuse for bundling libraries was that it helps them release faster.

On stability

Posted Sep 9, 2010 15:39 UTC (Thu) by ringerc (subscriber, #3071) [Link]

In truth I really like rapid releases - for desktop software on home machines, and in development tools/libraries. Bugs in software I use myself I can work around or fix, or I can roll back to an older version until the issue is fixed in a later release.

It's only when I have my sysadmin hat on and am responsible for the reliability of a network of machines used by other people that I start to want stability and time to fix things before the next update breaks everything all over again. Even then, I still really like rapid releases ... it's only when they're accompanied by the total abandonment of any support for any older releases that they bug me.

I do think FF and Chrome may be rushing into the future a little *too* fast - not in the sense of improving too rapidly, but in being unwilling to keep a release or two around for at least security fix purposes.

On stability

Posted Sep 9, 2010 21:54 UTC (Thu) by roelofs (guest, #2599) [Link]

As far as I had understood both "roelofs" and "ringerc" were complaining about release speed, the time a release is maintained, and new features bringing new bugs.

Yup.

AFAICT that has nothing to do with bundling of libraries.

To the extent the bundled libraries are modified, it certainly does. And even where they're not modified, the mere fact that they're additional copies means extra pain when security issues affect (or may affect) them--particularly when the browser version is no longer maintained by upstream.

But that discussion is more relevant to an earlier article.

Greg

On stability

Posted Sep 16, 2010 14:57 UTC (Thu) by robbe (guest, #16131) [Link]

The kernel community (Greg KH, with much help from others) do release updates to certain kernels, essentially creating upstream-supported stable branches.

I cannot see Firefox or Chromium upstream doing that. They seem to jettison support for older version once the new one is out.

Another difference is that the big distros seem to fund quite a bit of manpower working on the kernel. How many people have their Firefox/Chromium work paid for by a Linux distro?

On stability

Posted Sep 9, 2010 10:36 UTC (Thu) by djm (subscriber, #11651) [Link]

Hear are two good reasons for FF and Chromium's rapid release cycle: 1) fixing security problems (these are large and complex application by necessity). 2) Adding new features, since the web platform is still evolving fast.

The alternative to this, as you correctly identify, is Internet Explorer. A browser that substantially lags FF and Chromium in standards compliance, features and in which very serious vulnerabilities are not fixed _for years_ because of the engineering difficulties in doing so.

On stability

Posted Sep 13, 2010 16:19 UTC (Mon) by gerv (subscriber, #3376) [Link]

The evolution of the web platform. In the past fifteen months, Firefox has added at least the following major features:

* Web fonts
* Vast improvements in SVG support
* Major speed improvements for JavaScript (JITs and other magic)
* Out of process plugins, to stop Flash crashing your browser all the time (which required a major internal rearchitecture)
* Profile sync
* Large chunks of HTML5 and CSS3
* Rendering system changes to support mobile and GPU-accelerated graphics

You can't do this with small patches on top of a stable base.

Gerv

On stability

Posted Sep 13, 2010 18:03 UTC (Mon) by Trelane (subscriber, #56877) [Link]

They also added support for Cairo 1.10 in trunk last week. This brings a lot of improvements in the longer-term, but is trouble in the short (my build system now needs retweaking).

Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds