LWN: Comments on "Here comes another Microsoft-funded report"

Bias is evident

man_ls — Thu, 17 Nov 2005 13:32:17 +0000

Finally, don't trust "Herbert H. Thompson, Ph.D." or "Security Innovation" to give a fair or honest view of Microsoft's competitors since the report was rigged against Linux and had as much pro-Microsoft propaganda as a marketing brochure.

In the "research" paper (rather cheap propaganda) that comes with the study, the only mention of "free software" or "open source" is in the last paragraph, after the conclusions, which reads:

In our study, it is clear that component-based solutions typical of open source can be painful to deploy without the proper expertise and, with or without that expertise, painful to grow as system requirements evolve. More homogeneous, interoperable platforms that work as a system may offer less painful maintenance and react better to evolving requirements.

No need to read the study. Better waste your time visiting James A. Whitaker, Ph. D.'s pitiful Center for Information Assurance. Served by an IIS 5 on Windows 2000, it times out often; I found random broken links some hours ago; there are even corporate logos missing on the partners page. And Whitaker's expertise is in reliability? This guy is a complete fraud.

Clown? How so?

bojan — Thu, 17 Nov 2005 11:09:23 +0000

Sorry, I confused this guy with the others that did just that - upgraded the glibc and ended up with a broken system (diagram on page 29 of the report).

Clown? How so?

dark — Thu, 17 Nov 2005 10:32:41 +0000

The quote you quote shows the admin checking the SuSE site for newer packages, which is reasonable, and then not installing them because they have an incompatible glibc requirement. So he did exactly as you recommend.

Business requirements?

dark — Thu, 17 Nov 2005 10:28:34 +0000

How could "Upgrade the OS" be a business requirement? Especially in light of your paragraph about "Business isn't like home use"? If the new version of the OS includes a feature that you need, then that feature would be a business requirement; but then all ways of installing that feature should be valid.

And commercial stuff has the same ...

Wol — Thu, 17 Nov 2005 08:39:34 +0000

Our database (IBM U2) has exactly the same sort of thing. It says "requires Windows whatever", and sometimes, if you want to upgrade the database, you have to upgrade the underlying OS.

In other words, the study faults linux for something that EVERY software vendor requires, namely it's moaning that "software doesn't work if you ignore its pre-requisites". Doh!

Cheers,
Wol

As it turns out...

leonbrooks — Thu, 17 Nov 2005 06:32:24 +0000

...going from Mandriva 2005LE to Mandriva 2006.0 using the packaging system instead of an install medium involves a cross-glibc migration which should be automatically handled by URPMI and isn't.

This involves unpacking the new glibc RPM by hand, using cpio to dump it into the filesystem to get a working RPM, then installing the RPM again with --force to run all of the scriptlets. This took me about 15 minutes to sort out the first time, and about 3 each successive time. Big whoop-ti-do.

Add a mailing list manager? urpmi mailman -- ooh, that was hard. Upgrade the OS? urpmi.removemedia -a discards the old media, urpmi.addmedia --distrib rsync://mirror.pacific.net.au/mandriva/official/2006.0/i586 to define new media (if you happend to be on WAIX or PIPE in Australia), urpmi urpmi to make sure the packager is updated (this happens automatically in newer versions), then urpmi --auto-select to complete the job.

I'd have to know which specific data miner and search engine were involved in order to comment on those, but 2006.0 ships with ~12000 packages and is dead easy to rebuild RPMs for. This makes it likely that something suitable is included, and easy to rebuild anything necessary but not included.

So what happens is the test demands an app which hasn't been built for your version of MS-Windows. The simple answer is: you're stuffed. Completely. You might be able to reinstall MS-Windows to get a new version, but the odds are good that now something else won't work.

xoddam — Thu, 17 Nov 2005 03:05:04 +0000

> Some of the anti-Linux studies don't simply make things up, but
> instead focus in on known weaknesses in Linux. That does not mean
> the problems do not exist, it just means the developer community
> should also focus on the problems and make them go away ... In
> short, the distro didn't offer a version of the software they
> needed to do the job and they had to go find and install it
> themselves. Not necessarily a "Linux" problem, definitely a
> Novell/Suse problem.

Major changes like glibc upgrades (and minor ones such as mySQL upgrades)
are problems which *every* Linux distributor has successfully solved.
The artificial constraints of this study prevented the system
administrators from using the distributor's ready-made solution of simply
upgrading to the next release. Your argument (I know you're being
Devil's Advocate, but that doesn't excuse you) is therefore specious.

What a bunch of clowns

bojan — Wed, 16 Nov 2005 23:42:26 +0000

Sorry, this bit:

> and particularly not SLES (which I'm sure comes with its own way of organising dependencies)

Should have gone at the end of this sentence:

> I cannot but conclude that people in question were not all that experienced in administering an RPM based distro.

What a bunch of clowns

bojan — Wed, 16 Nov 2005 23:11:13 +0000

> For example, one administrator went to the SuSE site for the newer version for MySQL that shipped with SLES 9. This version, however, was built against a more recent version of GLIBC which then lead him to the MySQL site for the MySQL upgrade built against SLES 8s existing GLIBC version

This "one administrator" is quite obviously a clown. Never, _ever_, install binary RPMS from a newer version of the distro, especially if they depend on crucial system libs like glibc. I've broken RPM based systems with that approach _many_ times when I started with Linux and I've seen other people do it, thus learning valuable lessons. The bottom line is that _experienced_ administrators don't do things like this.

How about building new MySQL from source? Or getting the SRPMS and building that? There were a number of better solutions than forcing a new version of glibc onto an existing system, which is probably the biggest no-no in the RPM world, and particularly not SLES (which I'm sure comes with its own way of organising dependencies). I cannot but conclude that people in question were not all that experienced in administering an RPM based distro. I'm personally maintaining a yum repository for our RHEL4 systems with custom built RPMS that include around 50 packages (and growing) for i386 and x86_64 architecture each, ranging from bind and logrotate to Oracle clients and IBM/Sun JDKs. So, it ain't all that difficult.

In the end, the "business" bottom line is that this new version of MySQL was _unsupported_ on this particular version of SLES, period. So, if the "business requirement" was put to Windows administrators to install an unsupported version of third party software (e.g. something designed for Windows Vista only) on W2K or W2003, how would they solve that puzzle? They wouldn't - it would be impossible for them to do anything, because they would have no access to the source (not that they would know how or have the tools to compile it).

I think we should just print a lot of T-shirts that say: "Windows is infinitely better than Linux. Are you happy now?" and send them to Microsoft.

Executive Summary

mgb — Wed, 16 Nov 2005 20:38:22 +0000

That was too long without an executive summary:

EXECUTIVE SUMMARY

This study shows that if a large corporation pays enough money, someone can be found who will contrive a scenario in which he knew, or should have known, that the Linux participants would face artificial and unnecessary complexities not affecting the Windows participants.

There are two critical aspects to the study, both undisclosed:

1) The selection of products which would cause the complexities is not disclosed.

2) The reason for artificially precluding the Linux participants from using a natural solution - an earlier upgrade to SLES 9 or different products - is not disclosed.

The sample size of one PhD is not statistically significant.

tangaroa — Wed, 16 Nov 2005 20:37:50 +0000

Mike Bird says a lot of this in a lot fewer words than me, but I was already writing this so here's a second opinion:

Some of the anti-Linux studies don't simply make things up, but instead focus in on known weaknesses in Linux. That does not mean the problems do not exist, it just means the developer community should also focus on the problems and make them go away. Remember the Mindcraft study where the hardware selection aggravated inefficiencies in the Linux kernel? The developers didn't just ignore it, they fixed the problem and we got a much faster kernel in multi-interface situations.

If you were to check the report and dig through all the marketing crud, you might learn how Microsoft came to its conclusions. The problem for Linux was most likely with the "business requirements" that were landed on the admins every 3 months. The report says:

"Greater complexity of component dependencies which had to be tracked down was a key contributor to failure to meet business requirements for the Linux solution."

In short, the distro didn't offer a version of the software they needed to do the job and they had to go find and install it themselves. Not necessarily a "Linux" problem, definitely a Novell/Suse problem. The report says only one Linux admin was able to fulfil all of the "business requirements" and each managed to mangle their system into a different mess of incompatible dependencies.

Business isn't like home use. If your competitors are doing something that gives them a business advantage, you have to do it as well or do it better or you die. If Linux can't do it, it doesn't mean "don't do that 'cause Linux can't", it means "if you need to do that, don't use Linux." The validity of this report hinges on how much these business requirements are really common requirements of a modern business.

The report hypes Windows's integration where "core system functionality is incorporated with the operating system itself", but judging by how the Linux admins reacted, they were told to do things that were not core Linux functions. So yes, it looks like the test was rigged for Linux to look bad. Perhaps we can see that as an opportunity to improve upon Linux.

Looking further through the report, the business requirements were:

Adding a data mining utility
Adding a better search engine
Adding a mailing list manager
Upgrade the OS

The data mining tool they told the administrators to install required an upgrade from MySQL 3, which came with Suse 8, to MySQL 4 which the report says voids the Novell support contract, and the report has the gall to fault Linux for this. Two of the admins blew their system apart by making the required glibc update, and a third had to custom compile several packages to keep his system running. Conclusion: "Linux" was set up to fail, although it would have been nice for Novell to have had MySQL4 packages. The search engine tool chosen by the reports' authors also required a glibc upgrade, with similar results. Again, Linux was set up to fail.

Conclusions to gain from this? Mine, take it or leave it: Distro maintainers should do more to make sure third-party upgrades work with their system and vice versa, since it was the distro's relationship with newer versions of third-party software that was at fault. Also, someone should figure out how to make core libs like glibc and libstdc++ (which broke my system's KDE on an upgrade a few years back) upgrade painlessly. Finally, don't trust "Herbert H. Thompson, Ph.D." or "Security Innovation" to give a fair or honest view of Microsoft's competitors since the report was rigged against Linux and had as much pro-Microsoft propaganda as a marketing brochure.

An hour's analysis:

mgb — Wed, 16 Nov 2005 20:12:29 +0000

An hour's analysis:

1) The study pitted experienced Linux admins against experienced Windows admins - except the test plan required that the Windows admins had to have twice as much experience on the particular versions as the Linux admins [p3fn1, pp42-43]. There was no screening to ensure that the respective groups of admins had similar experience in the new applications to be installed during the study [pp41-44].

2) The study arbitrarily redefined reliability to mean "ease of adding new functions" [p2]. The study assumes that a new functional requirement comes along every 3 months [p3] and the IT department has only 10 hours to implement the function [p4]. The study then asked both teams to implement 3 new functions [p6, p22], 2 of which coincidentally [NOT - see (10) below] involved difficult glibc issues. When 2 of 3 Linux admins were unable to pull together all the package dependencies in 10 hours, the 14 unresolved dependencies were spun as "14 system failures" [p4].

3) The study arbitrarily limits security patches to one batch per month, to match Microsoft's cycle, as otherwise Suse would have patched problems much earlier [p13, p20].

4) The study reports that participants had to use MySQL patches from the MySQL website and backported glibc patches [p25], apparently in order to support the new search product chosen by the study designers [p26].

5) The study's conclusions largely derive from the pain we encounter during a glibc transition [pp28-31]. This may be fair criticism. Although a more accurate comparison would have been Windows NT to Windows 2000, glibc transition incompatibilities do occur more often than major Windows updates.

6) On the other hand, the study complements Suse on pulling everything back to normal with the transition to SLES 9 [p32]. Experienced admins, without the constraints of this study, would most likely have transitioned to SLES 9 when the new glibc requirement became apparent, rather than holding off for nearly a year so that the test case could be made artificially and unnecessarily perverse. This would also have obviated the need for manual patch management rather than using YAST [p34].

7) The study unfavorably reviews the larger number of patches applied to Suse [p4, pp26-27] without considering the size of the patches or the likelihood that code subject to world-wide review is likely to be more secure than code that daren't face the light of day.

8) The study does not report the number of reboots required for Windows patches, although it mentions in passing that a larger proportion of Windows patches required a reboot [p33]. Curiously the participants were required to log reboots [p39], the study just chose not to report them [pp26-27].

9) The nature of the log form clearly indicates that the upgrade difficulties were anticipated by the study designers [pp39-40]. We know that such situations occur, but rarely. Consequently, it is clear that the study designers chose a scenario which would run into glibc problems, precluded the participants from taking the obvious solution of a timely upgrade to SLES 9, and then gloated over the resulting mess.

10) Although the new functions to be implemented were reportedly best of breed and supported on both Windows and Linux, the study refuses to identify them [p45]. Thus we don't know whether the products have similar levels of support on both platforms, or whether better solutions could have been deployed if the Linux admins were not limited to deploying products chosen by Microsoft's flunkies and which had known glibc issues.

There may be mistakes in this analysis. Microsoft doesn't fund me so I could only spare an hour. Hope this helps.

--Mike Bird

Here comes another Microsoft-funded report

man_ls — Wed, 16 Nov 2005 18:54:38 +0000

The paper is in the reliability section.

You have a "research" paper and the actual study. The paper must be the dumbest thing ever published; it contains ground-breaking sentences like:

Other sources of downtime include vulnerabilities, load and functional bugs: all sources of failure, downtime and expensive recovery.

Emphasis mine. Or a list of bullets like this:

A typical installation includes the following:

A web server
[... some things ...]
Ability to handle html and http
[... other things ...]

Read it for a cheap laugh. Actually not even that; it gets boring around page 9.

It is published by Security Innovation, Inc.; but the subject has little to do with security. It seems that Microsoft has run out of prestigious consulting houses to sink in misery, and now has to turn to outfits with no prestige whatsoever -- and few chances of gaining any.

Here comes another Microsoft-funded report

flewellyn — Wed, 16 Nov 2005 18:27:28 +0000

Yeah, no kidding. Of course, Microsoft here is aiming specifically at "Linux", by which they mean GNU/Linux (comparing kernels alone is rather difficult), but their general target is free software versus their proprietary offerings.

It's only one data point among many, but one of my tasks as a sysadmin this past year was to replace our old MS SQL Server database with a PostgreSQL installation. I work for a GIS company that has moderately heavy database load, and we need reliability and good performance.

SQL Server, of course, is Microsoft's "enterprise" RDBMS, compared to Access, which is more "entry level". What this turns out to mean is that, really, the choice between Access and SQL Server is a trade-off: do you want a small feature set, relatively easy administration, low licensing fees, and piss-poor reliability, or do you want a large feature set, difficult and complex administration, very high licensing fees, and piss-poor reliability?

Just as a means of comparison: our SQL Server installation crashed, and had to be restarted, an average of once every two to three days. When the migration to PostgreSQL was complete, average uptime was...well, actually, it's only been taken offline once since then, for an upgrade from 7.4 to 8.0. In reliability alone, the free software approach won big. As for performance, features, flexibility, and administrative complexity...it's no fair comparison at all. SQL Server is left in the dust.

Here comes another Microsoft-funded report

carcassonne — Wed, 16 Nov 2005 17:13:14 +0000

"The current white papers seem to mostly assume that their readers are stupid or ignorant (this is not limited to Microsoft). "

No surprise: they're generally aimed at decision makers and other managers.

;-)

Here comes another Microsoft-funded report

jmtapio — Wed, 16 Nov 2005 16:17:45 +0000

It seems ridiculous to take a sample of two and make any kind of
conclusions based on the results (for or against Linux), but I guess that
is to be expected from Microsoft.

It would be fun to look into their methodology but I can not seem to find
the actual paper. The press release points to the Get the Facts -page but
I do not see the paper there. Am I blind or in the wrong time zone or
something?

I would like to see at least once in my lifetime a marketing white paper
that would use proper research methodology. The current white papers seem
to mostly assume that their readers are stupid or ignorant (this is not
limited to Microsoft).

Here comes another Microsoft-funded report

smitty_one_each — Wed, 16 Nov 2005 14:55:26 +0000

At least the source is clearly marked on the propaganda.

Here comes another Microsoft-funded report

wilreichert — Wed, 16 Nov 2005 14:53:52 +0000

"Our research indicates that the primary methods of computing reliability as indicators of real IT pain are overly simplistic."

Heh.