LWN.net Logo

win2003 v Linux bias

From:  Richard Kay <rich@copsewood.net>
To:  mike.magee@theinquirer.net
Subject:  win2003 v Linux bias
Date:  Wed, 7 May 2003 12:57:42 +0100
Cc:  letters@lwn.net

In article http://www.theinquirer.net/?article=9333 you wrote:
 
>IT WILL COME as no surprise that the tests involved, although performed
>by an independent lab, were sponsored by Microsoft. But that doesn't
>change their basic validity. The tests were all about performing that
>most basic of tasks, file serving. And Windows thrashed Linux.
 
Sorry but this does significantly compromise their basic validity.
 
First of all it would appear that Microsoft chose the systems on
which the testing lab carried out the
comparison , and the Red Hat system seems to have been running
a 2 year old Kernel. The latest Linux stable kernel (2.4.19 ?) is
now lacking about a years worth of SMP performance work on the
development series (2.5.x) while the Linux development series is probably
not yet stable enough in its product cycle for a fair comparison.
 
Secondly I would have thought that it is common knowledge that
what Microsoft pays for is by definition not independent. The
testing firm claims to have investigated performance issues
concerning the Samba configuration, but was there any open
invitation to the Samba development community to recommend configuration
options for the setup, or to debug problems with this ? Presumably
Microsoft's leading system developers were involved in the
choice and configuration of the Windows 2003 version.
 
Thirdly the test involved using a network file-sharing protocol which
is of Microsoft's design, which Microsoft has chosen not to publish
full documentation for. The fact that previous versions of Samba have
outperformed previous versions of Windows is testament to the reverse
engineering skills of the Samba developers, but when the choice of
playing field is so clearly sloping in one direction this is hardly
comparing like with like. How about a comparison involving a
Microsoft system chosen and setup by Linux NFS developers serving
NFS clients in competition with Linux ?
 
Sincerely,
Richard Kay
rich@copsewood.net


(Log in to post comments)

win2003 v Linux bias

Posted May 8, 2003 2:53 UTC (Thu) by mbcook (subscriber, #5517) [Link]

I agree quite a bit with the first point. I can understand not running a 2.5 kernel (it's not officialy stable, so it wouldn't be what you'd run in a production environment), but to run a 2 year old version of the stable kernel is just sloppy and might largly invalidate the test. Comparisions need to be close to fair if you want respect.

I agree with the second point as well. How different would the results be if all the hardware was picked to perform better under Linux than Windows. That said, I agree with the first sentence: any test that is paid for by one of the compeditors is, by definition, not independent.

It's the third point here that has bugged me the most. Sure Linux is being forced to compete on foreign soil, but there is something else there. The big difference is the kernel. A computer running Linux must, of course, run Samba as well. A computer running Windows doesn't need a second program like Samba, the code is probably IN THE KERNEL where is has a significant advantage (speed wise) over userspace code (like Samba). The trade-off is security (which Samba/Linux has chosen) versus speed (Windows). When it comes down to it Windows will, more or less, always be faster than Samba because Samba is always playing catchup. Samba was only faster in the past because MS got lazy and didn't bother trying to improve things much (this is the conclusion I draw, it seems the most logical).

For things to be fair, you'd need to use a filesystem that isn't native to either OS. The letter seems to state that we should compare the speed of Windows serving Windows shares to Windows comptuers to Linux serving NFS shares to other Linux computers. This would be an interesting comparison, but I think it would be more important to see the speed of sharing NFS shares to a GROUP of OSes (taking the fastest scores) so that we can be sure that the client isn't effecting the test, only the server.

That said, I would like to say Kudos to MS for improving their products. For all the griping that many (including myself) do, it's great to see that they have made a product that runs faster than the previous two version (NT4 and NT5/2k) on THE SAME HARDWARE. It's one thing to say that your new version can serve data faster (by a few percent) than the previous version (but you need faster hardware to squeeze a measureable about out of it); but it's another to give free performance to people (50% or more) without requiring "fast" hardware. This shows us two things: MS can improve things when they set their mind to it, and competition (Linux/Samba) is good for everyone. Without Samba, would MS have improved things so much? Maybe, but having competition makes it much more likely that they would.

win2003 v Linux bias

Posted May 8, 2003 20:36 UTC (Thu) by Peter (guest, #1127) [Link]

I can understand not running a 2.5 kernel (it's not officialy stable, so it wouldn't be what you'd run in a production environment)

Yeah. MS took advantage of release cycles here. If they had waited 6 months to do their test, they might have had to run against Linux 2.6. (OK, so I'm being optimistic....) As it is, they are running against the current stable Linux release, so you can't really complain. (Except for the small matter of not applying any Red Hat updates, like newer kernels, of course.)

It's the third point here that has bugged me the most. Sure Linux is being forced to compete on foreign soil, but there is something else there. The big difference is the kernel. A computer running Linux must, of course, run Samba as well. A computer running Windows doesn't need a second program like Samba, the code is probably IN THE KERNEL where is has a significant advantage (speed wise) over userspace code (like Samba).

Well, actually, NT (presumably including w2k3) uses a microkernel approach, so most likely the SMB file sharing is actully in userspace. I don't know for sure. But that is beside the point. The point is, it would be possible to put a SMB file server in the Linux kernel - there has been past discussion about a "samba accelerator" kernel module - it just hasn't been done.

Also, SMB is not "native" to Windows any more than NFS is "native" to Linux. Certainly Microsoft has had more influence on the development of the CIFS standard than anyone else, and they've certainly tailored their file server offerings around it to an extent, but it is still an open standard. In fact, recently Samba has started to offer non-Microsoft extensions to SMB, for a few Unixy features.

but it's another to give free performance to people (50% or more) without requiring "fast" hardware.

You're not talking about this benchmark anymore, then, right? Because this one was done with an 8-way Xeon. I dunno, that seems like "fast" hardware to me. (:

win2003 v Linux bias

Posted May 8, 2003 20:47 UTC (Thu) by mbcook (subscriber, #5517) [Link]

That was an 8 way xeon wasn't it. Oops, forgot that. Silly me.

win2003 v Linux bias

Posted May 9, 2003 19:44 UTC (Fri) by dbreakey (guest, #1381) [Link]

Windows has used a microkernel approach since Windows NT so, presumably, Win2k3 also utilizes that approach. However, the chances are that the SMB protocol code is allowed to execute within the privileged ring which, for all practical purposes, makes it part of the kernel. The SMB code is "presumed" to be trustworthy. This has been documented in several of the Resource Kit books, so I feel reasonably confident with this conclusion. There is a great deal of code that operates in this fashion, even though it is technically not part of the kernel itself.

And SMB is effectively native to Windows, since it is the only major operating system that still utilizes it as a "default" networking environment. Current versions of Windows support TCP/IP, and even do so effectively, but it's still clear that SMB is a key part of their product, if for no other reason than backwards compatibility.

Also, the entire CIFS protocol was originally developed by Microsoft, and then released as an open standard. And if you read the article, it is fairly clear that CIFS was based on the SMB implementation present in Windows. And since Microsoft is effectively considered the authoritative implementor of the SMB protocol, they undoubtably still hold a great deal of influence over how the CIFS protocol has evolved.

Additionally, which SMB may not have originally been a Microsoft development (I can't find evidence either supporting or denying this, but I've only performed a basic search), it is clearly strongly associated with the Windows operating system. In fact, the only other OS that I know of that implemented SMB "by default", so to speak, was IBM's OS\2, due to the common heritage that NT and OS\2 originally shared.

Finally, even though Microsoft claims to fully support the CIFS standard, they have never, to my knowledge, commited to using it in preference to their own internal SMB implementation; and they certainly have never commited to "never again" modify and extend the capabilities of that same SMB implementation.

win2003 v Linux bias

Posted May 10, 2003 11:28 UTC (Sat) by Cato (subscriber, #7643) [Link]

Windows NT/2000/XP/2003 can't really be described as using a true microkernel approach - there is a small microkernel within the main kernel, but unlike true microkernels (e.g. Mach) you can't plug in user space processes that deliver 'kernel' services. Microsoft themselves use the term 'modified microkernel' or 'macrokernel' in this white paper:
http://msdn.microsoft.com/archive/en-us/dnarwbgen/html/msdn_movuser.asp

win2003 v Linux bias

Posted May 12, 2003 18:46 UTC (Mon) by dbreakey (guest, #1381) [Link]

Thanks for the clarification; I wasn't really sure whether it was a true microkernel or not; all I've personally read about it is what is present in the various resource kit guides.

I do know that various services, which should be userspace services, are permitted to run at an elevated level and are therefore permitted to interact with the kernel almost as if it were part of the kernel itself (in other words, while the service, technically, is not part of the kernel, it is considered trustworthy. For performance reasons, therefore, the Windows kernel is not as careful about checking requests and parameters, at least in regards to these select services).

I know that this is due to a necessary trade-off between performance and reliability, but Linux seems to manage this just fine. If it isn't, how is it that I can crash Windows 2000 (bluescreen) simply by interacting with a system-tray icon? I admit, this happens infrequently, but it shouldn't happen at all--at first, I thought it might be faulty hardware, but Linux has been running on the exact same box for months without so much as a hiccup. Therefore, I can only conclude that the software is messed up.

Ah well. I've now adopted a solution that seems to work perfectly (running Windows inside an emulated PC), so it's no longer an issue. So far …

win2003 v Linux bias

Posted May 8, 2003 19:50 UTC (Thu) by vonbrand (subscriber, #4458) [Link]

What where the benchmarks that _weren't_ published? What did they test, on what machines?

I did not read the results (yes, I'm lazy ;-), but was this a one-on-one fileserving test, or a bunch of clients? The second case is the only one that interests really. One big file, or lots of smallish ones (that is what I'd expect from Win clients accessing their stuff over the net)? "Hot cache" (i.e., 100MiB file over and over, on several GiB RAM) or not? One or several networks (again, one network is probably the most common real-world configuration)?

BTW, is there anybody out there using 32 way machines as fileservers?!

Besides all this, the results in _one_ (rigged or not, synthetic or real-world, ...) benchmark is just not enough to decide anything when contemplating the purchase of such massive hardware, this is clearly marketing aimed at the not-so-hard-looking prospective buyer of Win2k3 for modest use.

Copyright © 2003, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds