Here comes another Microsoft-funded report

Posted Nov 16, 2005 14:53 UTC (Wed) by wilreichert (subscriber, #17680) [Link]

"Our research indicates that the primary methods of computing reliability as indicators of real IT pain are overly simplistic."

Heh.

Posted Nov 16, 2005 16:17 UTC (Wed) by jmtapio (guest, #23124) [Link]

It seems ridiculous to take a sample of two and make any kind of
conclusions based on the results (for or against Linux), but I guess that
is to be expected from Microsoft.

It would be fun to look into their methodology but I can not seem to find
the actual paper. The press release points to the Get the Facts -page but
I do not see the paper there. Am I blind or in the wrong time zone or
something?

I would like to see at least once in my lifetime a marketing white paper
that would use proper research methodology. The current white papers seem
to mostly assume that their readers are stupid or ignorant (this is not
limited to Microsoft).

Posted Nov 16, 2005 20:12 UTC (Wed) by mgb (guest, #3226) [Link]

An hour's analysis:

1) The study pitted experienced Linux admins against experienced Windows admins - except the test plan required that the Windows admins had to have twice as much experience on the particular versions as the Linux admins [p3fn1, pp42-43]. There was no screening to ensure that the respective groups of admins had similar experience in the new applications to be installed during the study [pp41-44].

2) The study arbitrarily redefined reliability to mean "ease of adding new functions" [p2]. The study assumes that a new functional requirement comes along every 3 months [p3] and the IT department has only 10 hours to implement the function [p4]. The study then asked both teams to implement 3 new functions [p6, p22], 2 of which coincidentally [NOT - see (10) below] involved difficult glibc issues. When 2 of 3 Linux admins were unable to pull together all the package dependencies in 10 hours, the 14 unresolved dependencies were spun as "14 system failures" [p4].

3) The study arbitrarily limits security patches to one batch per month, to match Microsoft's cycle, as otherwise Suse would have patched problems much earlier [p13, p20].

4) The study reports that participants had to use MySQL patches from the MySQL website and backported glibc patches [p25], apparently in order to support the new search product chosen by the study designers [p26].

5) The study's conclusions largely derive from the pain we encounter during a glibc transition [pp28-31]. This may be fair criticism. Although a more accurate comparison would have been Windows NT to Windows 2000, glibc transition incompatibilities do occur more often than major Windows updates.

6) On the other hand, the study complements Suse on pulling everything back to normal with the transition to SLES 9 [p32]. Experienced admins, without the constraints of this study, would most likely have transitioned to SLES 9 when the new glibc requirement became apparent, rather than holding off for nearly a year so that the test case could be made artificially and unnecessarily perverse. This would also have obviated the need for manual patch management rather than using YAST [p34].

7) The study unfavorably reviews the larger number of patches applied to Suse [p4, pp26-27] without considering the size of the patches or the likelihood that code subject to world-wide review is likely to be more secure than code that daren't face the light of day.

8) The study does not report the number of reboots required for Windows patches, although it mentions in passing that a larger proportion of Windows patches required a reboot [p33]. Curiously the participants were required to log reboots [p39], the study just chose not to report them [pp26-27].

9) The nature of the log form clearly indicates that the upgrade difficulties were anticipated by the study designers [pp39-40]. We know that such situations occur, but rarely. Consequently, it is clear that the study designers chose a scenario which would run into glibc problems, precluded the participants from taking the obvious solution of a timely upgrade to SLES 9, and then gloated over the resulting mess.

10) Although the new functions to be implemented were reportedly best of breed and supported on both Windows and Linux, the study refuses to identify them [p45]. Thus we don't know whether the products have similar levels of support on both platforms, or whether better solutions could have been deployed if the Linux admins were not limited to deploying products chosen by Microsoft's flunkies and which had known glibc issues.

There may be mistakes in this analysis. Microsoft doesn't fund me so I could only spare an hour. Hope this helps.

--Mike Bird

Posted Nov 16, 2005 20:37 UTC (Wed) by tangaroa (guest, #33897) [Link]

Mike Bird says a lot of this in a lot fewer words than me, but I was already writing this so here's a second opinion:

Some of the anti-Linux studies don't simply make things up, but instead focus in on known weaknesses in Linux. That does not mean the problems do not exist, it just means the developer community should also focus on the problems and make them go away. Remember the Mindcraft study where the hardware selection aggravated inefficiencies in the Linux kernel? The developers didn't just ignore it, they fixed the problem and we got a much faster kernel in multi-interface situations.

If you were to check the report and dig through all the marketing crud, you might learn how Microsoft came to its conclusions. The problem for Linux was most likely with the "business requirements" that were landed on the admins every 3 months. The report says:

"Greater complexity of component dependencies which had to be tracked down was a key contributor to failure to meet business requirements for the Linux solution."

In short, the distro didn't offer a version of the software they needed to do the job and they had to go find and install it themselves. Not necessarily a "Linux" problem, definitely a Novell/Suse problem. The report says only one Linux admin was able to fulfil all of the "business requirements" and each managed to mangle their system into a different mess of incompatible dependencies.

Business isn't like home use. If your competitors are doing something that gives them a business advantage, you have to do it as well or do it better or you die. If Linux can't do it, it doesn't mean "don't do that 'cause Linux can't", it means "if you need to do that, don't use Linux." The validity of this report hinges on how much these business requirements are really common requirements of a modern business.

The report hypes Windows's integration where "core system functionality is incorporated with the operating system itself", but judging by how the Linux admins reacted, they were told to do things that were not core Linux functions. So yes, it looks like the test was rigged for Linux to look bad. Perhaps we can see that as an opportunity to improve upon Linux.

Looking further through the report, the business requirements were:

• Adding a data mining utility
• Adding a better search engine
• Adding a mailing list manager
• Upgrade the OS

The data mining tool they told the administrators to install required an upgrade from MySQL 3, which came with Suse 8, to MySQL 4 which the report says voids the Novell support contract, and the report has the gall to fault Linux for this. Two of the admins blew their system apart by making the required glibc update, and a third had to custom compile several packages to keep his system running. Conclusion: "Linux" was set up to fail, although it would have been nice for Novell to have had MySQL4 packages. The search engine tool chosen by the reports' authors also required a glibc upgrade, with similar results. Again, Linux was set up to fail.

Conclusions to gain from this? Mine, take it or leave it: Distro maintainers should do more to make sure third-party upgrades work with their system and vice versa, since it was the distro's relationship with newer versions of third-party software that was at fault. Also, someone should figure out how to make core libs like glibc and libstdc++ (which broke my system's KDE on an upgrade a few years back) upgrade painlessly. Finally, don't trust "Herbert H. Thompson, Ph.D." or "Security Innovation" to give a fair or honest view of Microsoft's competitors since the report was rigged against Linux and had as much pro-Microsoft propaganda as a marketing brochure.

Posted Nov 16, 2005 23:11 UTC (Wed) by bojan (subscriber, #14302) [Link]

> For example, one administrator went to the SuSE site for the newer version for MySQL that shipped with SLES 9. This version, however, was built against a more recent version of GLIBC which then lead him to the MySQL site for the MySQL upgrade built against SLES 8’s existing GLIBC version

This "one administrator" is quite obviously a clown. Never, _ever_, install binary RPMS from a newer version of the distro, especially if they depend on crucial system libs like glibc. I've broken RPM based systems with that approach _many_ times when I started with Linux and I've seen other people do it, thus learning valuable lessons. The bottom line is that _experienced_ administrators don't do things like this.

How about building new MySQL from source? Or getting the SRPMS and building that? There were a number of better solutions than forcing a new version of glibc onto an existing system, which is probably the biggest no-no in the RPM world, and particularly not SLES (which I'm sure comes with its own way of organising dependencies). I cannot but conclude that people in question were not all that experienced in administering an RPM based distro. I'm personally maintaining a yum repository for our RHEL4 systems with custom built RPMS that include around 50 packages (and growing) for i386 and x86_64 architecture each, ranging from bind and logrotate to Oracle clients and IBM/Sun JDKs. So, it ain't all that difficult.

In the end, the "business" bottom line is that this new version of MySQL was _unsupported_ on this particular version of SLES, period. So, if the "business requirement" was put to Windows administrators to install an unsupported version of third party software (e.g. something designed for Windows Vista only) on W2K or W2003, how would they solve that puzzle? They wouldn't - it would be impossible for them to do anything, because they would have no access to the source (not that they would know how or have the tools to compile it).

I think we should just print a lot of T-shirts that say: "Windows is infinitely better than Linux. Are you happy now?" and send them to Microsoft.