Matt Domsch on Linux support at Dell [LWN.net]

By Jake Edge
October 3, 2007

Curious about the state of Linux support at Dell, we asked Matt Domsch, Linux Technology Strategist there, some questions by email. Matt has been working to support Linux on Dell hardware for eight years and has broad knowledge of the kinds of issues hardware vendors face when supporting Linux. He was gracious enough to answer our questions at length, covering some history, challenges and the future for Linux at Dell.

LWN: Can you give us an overview of your Linux history? When did you get involved with Linux and why?

While I had seen a few PCs running Linux back in 1993 and 1994 while I was at MIT, it wasn't until I started at Dell in 1994 that I had access to PCs of my own that I could use for Linux work. I immediately installed Slackware on my "Engineering Workstation", a 486DX2-class system, and rebuilt kernels regularly for about a year. I even used Linux as a network stress-test tool for a while. I was a prototypical early Linux user, not yet a developer.

LWN: What is your history at Dell? And the history of Linux support at Dell?

In 1999, Dell was receiving a lot of customer requests for Linux to be pre-installed on our servers. These requests came through our Custom Factory Integration team, who could install OS images, but who weren't set up to support Linux as a product. Dell saw the volume of these requests and decided to make Linux a formal product offering on our PowerEdge servers. I was just finishing a Master's degree at Vanderbilt and working for Dell when I was asked to help with "writing and fixing some device drivers". Little did I know it would become my life for the next 8 years as lead engineer then architect for the team.

Shortly thereafter, because sales were really taking off, we made Linux a "Tier 1" operating system on PowerEdge servers. Tier 1 means we won't ship a server to anyone until Linux, in particular our Linux partner of choice then, Red Hat Linux, ran well on the system. Previously, only Microsoft Windows and Netware had this distinction. In this change, we moved from a relatively small number of people developing and testing on Linux, to hundreds of engineers, software developers, and testers across the company working together to deliver stable products. The test organizations, OpenManage systems management software teams, groups that interact with our partner suppliers such as Intel, Adaptec, LSI, Broadcom, and the like, all stepped up to make sure their components were well supported.

We sold Red Hat Linux, starting with version 6.1 all the way through 9, then Red Hat Enterprise Linux 2.1 to present, on our PowerEdge servers. Along the way we added Precision Workstations to the mix. In 2004 we added Novell/SuSE Linux Enterprise Server on the servers, and in 2007, because of the strong demand seen on Dell IdeaStorm, we added Ubuntu on select desktop and notebook systems. In these 8 years, I've helped build a strong team of engineers who deliver these products, cycle after cycle.

I've now progressed to be a Linux Technology Strategist in Dell's Office of the CTO, helping to set Dell's future directions with Linux. In addition, I'm active in the Fedora Project a member of the Fedora Project Board.

LWN: What have been the biggest challenges with supporting Linux at Dell?

Release schedules, and particularly, the dichotomy between new hardware release schedules, and schedules for 3rd party Independent Software Vendors (ISVs).

With Red Hat Linux, we were churning every 6 months. That works great for new hardware - there were lots of opportunities get new systems supported in the next release; but that churn is way too fast for ISVs. What we found is that if people were only using software that was included in the distribution, they didn't mind the frequent upgrades. But as soon as they had a dependency on a 3rd party application, the upgrade schedule just couldn't work, so they'd run older, less-supported versions of their distributions for longer. Companies like Red Hat and Novell/SuSE figured this out, which is why they've extended the time between major version releases, and extended each version's supported life cycle.

So now we've got longer stretches of time between OS releases, but the hardware still comes out on its own quicker schedule. We created DKMS - Dynamic Kernel Module Support, as our tool to help fill this gap. DKMS lets us update individual kernel modules (device drivers), without replacing the whole kernel. In the Dell factories, this lets us keep the widely tested "Gold" kernel for a given OS version, and replace just the specific kernel modules we must to fix major bugs and enable newer hardware. We also ensure that these fixes are rolled into the next Service Pack or scheduled Update from the OS vendor.

[PULL QUOTE: Dell expects our partner hardware vendors to provide fully open source / free software drivers, and to maintain them in the appropriate upstream project, kernel.org or x.org. This has been our policy since we started shipping Linux on servers in 1999, and continues to be our policy today. END QUOTE] LWN: Does Dell have plans to make hardware changes in the future to support more hardware with free drivers, specifically video and wireless cards?

Dell expects our partner hardware vendors to provide fully open source / free software drivers, and to maintain them in the appropriate upstream project, kernel.org or x.org. This has been our policy since we started shipping Linux on servers in 1999, and continues to be our policy today. While we have a few holdouts, I praise the work Intel has done for their wireless and video drivers, and have high hopes for AMD/ATI video drivers given their recent announcements that they are working with the community to develop and maintain fully open source drivers.

LWN: You folks recently made a new version of Ubuntu with updated drivers and the like, can you give us an idea of what the aim of that effort was? What kind of problems were you trying to solve and what plans do you have to work with Ubuntu to avoid that in the future?

This really comes back to the timing of releases again. When we started working with Ubuntu/Canonical this Spring, the Ubuntu 7.04 release was in beta, and already supported most of the hardware we needed for the initial product offering. But we knew we had newer platforms coming down the pipe that we wanted to pre-install Ubuntu on, and 7.04 didn't have drivers capable of working on that new hardware yet. DKMS helps, but if you're missing a storage driver, a network driver, and a video driver from the installation CD, it's still hard for a user to get an OS installed on that system. We produced the remastered Ubuntu CDs exactly to make it easy for users to install Ubuntu 7.04 on these newer systems.

We try our best to get future hardware supported in upstream kernel.org and x.org, and therefore in the distributions, as early as possible. The drivers we missed in Ubuntu 7.04 were included in upstream in time to be ready for Ubuntu 7.10's kernel freeze. Depending on the distro's release schedule, we may have to have code in upstream a year ahead of actual hardware availability to customers, and that's really difficult when the hardware itself churns faster than this. In parallel with the upstream submissions, we work with the distro teams to get those same drivers added.

LWN: Can you tell us about the testing and support lab at Dell?

Dell has hundreds of engineers and technicians around the globe who develop software for Linux, and test Linux our systems. We develop Linux for PowerEdge servers, Precision workstations, and select desktop and notebook systems.

The primary Linux Engineering team has two centers, one in Austin, Texas, USA and one in Bangalore, India, but it is one cohesive team. We run Linux on our own systems for day-to-day use. In addition to the Engineering team, Dell's Enterprise Test organizations put considerable effort into ensuring Linux "just works" on our systems. Our Technical Support organization has special teams of people with significant system administration and debugging experience, to handle customer issues. Many of these people have Linux certifications such as Red Hat Certified Engineer.

In addition to driver development and testing, we develop open source projects such as DRU (Disc Remastering Utility for Ubuntu), libsmbios, firmware-tools, biosdevname, and DKMS. These are available on linux.dell.com.

LWN: What is DKMS? What's the status of the project? What still remains to be done?

Noted above, DKMS is our tool that lets us provide an updated device driver without replacing the entire kernel. We use it as a stopgap mechanism, and work with our OS distribution partners to get those updates included into future distro-provided kernels.

DKMS leverages the kernel's own Kbuild system for compiling drivers from source code. It can handle lots of drivers, multiple versions of each, built for multiple architectures. It has a mode, the 'autoinstaller', where it can recompile driver source code on an end user's system when they upgrade their kernel. DKMS recognizes when the kernel's copy of a device driver is the same or newer than one provided elsewhere, so it can stop using the out-of-tree driver as soon as the in-tree driver is "good enough".

DKMS is extremely handy for testing changes to device drivers as you prepare it for inclusion in kernel.org, and for out-of-tree drivers who have not yet completed their task of being included in kernel.org.

DKMS is widely used in Mandriva and PCLinuxOS, and included in Fedora and Ubuntu. DKMS has been quite stable for while, accomplishing its primary goal. Recently we've enhanced it to add Ubuntu support - it can now create deb packages out of device drivers just as it makes RPMs and Red Hat / Novell Kernel Module Packages (KMPs). It can make driver disks for all of these distros too.

We also recently added a way to make it easy to download new driver RPMs specifically for your hardware. With a little more work, you will be able to "yum install" updated drivers when they're released by Dell.

As for the future, I see DKMS growing features to help driver developers, such as building driver disks for more distributions, and building better, more easily consumable KMPs. I'd like to see it included in more distributions, but I don't expect any huge changes - it does its job well.

LWN: Where do you see Linux at Dell heading? More systems supported, more distributions supported, or other things entirely?

While our sales of desktops and notebooks with Ubuntu pre-loaded have met our expectations, I personally hoped that Ubuntu sales would be through the roof. In August we expanded sales from US-only into Germany, France, and the UK, so we're just starting to see the sales there. We're working to add the next version of Ubuntu, 7.10, on the existing set of notebooks and desktops. What additional systems get offered depends on sales.

As for the number of distros supported, Dell already sells Red Hat Enterprise Linux on our PowerEdge servers and Precision workstations, Novell/SuSE Linux Enterprise Server on our PowerEdge servers, Ubuntu on our desktops and notebooks, and we've announced plans to add Novell/SuSE Linux Enterprise Desktop on notebooks and desktops in China. That's a pretty wide coverage already.

One challenge we'll continue to have is the sheer number of Linux distributions available that we could offer on our systems. Clearly we won't validate and sell all of them. Our method for combating the proliferation of distributions is to work with the upstream projects to get our hardware supported, and then to let each distribution pull from those upstream projects. This policy let us add Ubuntu to our product line in record time - it would have taken much longer had we needed to write and validate all the drivers for those systems from scratch. This policy also lets users choose which distro they run based on their business needs or personal preferences, not solely on what Dell chooses to validate.

A second strategy we're pursuing which should help is virtualization. If instead of testing each system/distribution combination, we could test a smaller number of system/hypervisor combinations, and the distros themselves could test their own hypervisor/distro combinations, then we get the best of both worlds - end user choice of distributions, with the same or lower development effort needed by each hardware manufacturer.

We're also going to continue looking for ways to make it easier, faster, and less expensive to bring new products to market. If we had waited to train phone personnel in all our offices around the globe, that would have been slow and expensive. With Ubuntu we've seen that online support offerings work well for most users, and that they don't want to pay for telephone-based support they don't feel they need. For us, online support methods are quicker and easier to develop, and they scale to many more users far better too. By delivering an appropriate level of support for these users, everyone saves time and money. Our public mailing lists, forums, and web pages at linux.dell.com do exactly this. And for those who want paid-for telephone support, we offer that as well.

One last plug. If you are buying a sufficient number of systems, Dell can do anything you need. Your custom-crafted Linux OS installed on a system to make a hardware appliance - no problem. Custom-crafted servers to fill your data centers - no problem. You won't find these offered in the Sunday newspaper, but we can do it.

Our thanks to Matt for taking the time to respond in detail.

(Log in to post comments)

Matt Domsch on Linux support at Dell

Posted Oct 5, 2007 14:47 UTC (Fri) by mdomsch (subscriber, #5920) [Link]

New machines are plenty fast for most applications/users such that the slight (and getting smaller all the time) performance loss to a virtualization layer is worth it for many people. It has little to do with proprietary drivers, or DRM. It's about the benefits virtualization can bring: application separation (for security, dependencies, resource limiting); live migration; ability to run multiple OSs concurrently; splitting of CPU and memory from storage (important for data centers with lots of storage on a Storage Area Network of any type). If we can leverage this same technology to increase the number of systems running Linux, and to increase the number of Linux distributions that can run on these systems, all the better.

The point of virtualization

Posted Oct 5, 2007 16:14 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

The virtualization layer doesn't slow the machine down, anyway. On a system built for virtualization, with only one virtual machine, you shouldn't notice a difference vs running directly on the real machine.

"Are machines too fast these days" can also be taken to mean, "are they so fast we need to run multiple systems on them to take advantage of all the cycles?" But that's not the case -- the speed of the machines follows demand, not the other way around.

In listing the advantages of virtualization, you need to say compared to what. All the advantanges listed are compared to running multiple applications under a single traditional opperating system image. But a more important comparison is to the way people actually achieve all those things today: run the separate applications under separate operating systems running on separate machines.

The advantage of virtualization over that is primarily management cost. People prefer to move memory from one machine to another by clicking a mouse rather than by taking apart the systems and moving circuit boards.

On the other hand, with pre-virtualization technology, they actually prefer to manage racks of physically separate machines to managing multiple applications on one.

One other technology along the continuum to keep in mind is blade servers: multiple operating system images on sort of one machine and sort of multiple machines.

The point of virtualization

Posted Oct 10, 2007 1:12 UTC (Wed) by roelofs (subscriber, #2599) [Link]

...until you get a TLB miss, anyway; then you pay double. See Ulrich's currently running series for details. ;-)

Greg

The point of virtualization

Posted Oct 10, 2007 16:13 UTC (Wed) by giraffedata (subscriber, #1954) [Link]

The virtualization layer doesn't slow the machine down, anyway. On a system built for virtualization, with only one virtual machine, you shouldn't notice a difference vs running directly on the real machine.
...until you get a TLB miss, anyway; then you pay double. See Ulrich's currently running series for details. ;-)

I believe on a system built for virtualization, with only one virtual machine, a TLB miss costs the same, and is equally likely, as on an equivalent system without a virtualization layer.

Of course, if you have a counterargument more specific than a 100 page tome, I might admit I'm wrong.

The point of virtualization

Posted Oct 11, 2007 5:09 UTC (Thu) by roelofs (subscriber, #2599) [Link]

Of course, if you have a counterargument more specific than a 100 page tome, I might admit I'm wrong.

Let the healing begin! :-)

Memory part 3: Virtual Memory

Specifically, see the section discussing shadow page tables: without some of the upcoming hardware assists, you need to do two "physical to virtual" translations (within the context of the given kernel--i.e., a "physical" address for a guest kernel isn't truly physical), and that involves (on x86-64) four separate, sequential, non-hardware-prefetchable memory accesses each--for a total of eight--plus the memory access for the actual data. IOW, 5 accesses become 9, and each one is extremely costly in terms of clock cycles due to the no-prefetch issue.

ObCaveat: this is all from Ulrich Drepper's paper and talk (recently given at a local LUG). While I'm pretty sure I've summarized it accurately, he's the expert, not me; I might be misrepresenting some of it. Also keep in mind that the really bad TLB penalties kick in for certain (large) data-member and working-set sizes, and neither of us has made any statement (that I'm aware of, anyway) as to how typical such things are in real applications. (For the kind I mostly work on, I'm pretty sure they have the potential to be quite common. Actually quantifying it, however, is...not so easy.)

Greg

The point of virtualization

Posted Oct 12, 2007 17:38 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

you need to do two "physical to virtual" translations (within the context of the given kernel--i.e., a "physical" address for a guest kernel isn't truly physical), and that involves (on x86-64) four separate, sequential, non-hardware-prefetchable memory accesses each--for a total of eight--plus the memory access for the actual data. IOW, 5 accesses become 9, and each one is extremely costly in terms of clock cycles due to the no-prefetch issue.

Ulrich doesn't mention physical to virtual translations. I presume you meant virtual to physical, since that's what would involve a TLB miss. Ulrich points out only one of those: the one through the shadow page table. The virtual page table is not involved in page translation. So it's the same speed with virtualization as without.

Leaving the topic of TLB misses, Ulrich does mention in this section that page table updates are more expensive, but I don't think that makes a noticeable dent in virtual machine speed. And he doesn't mention that some systems built for virtualization reduce that by doing away with the duplicate page tables.

without some of the upcoming hardware assists,

Those are probably included in my phrasing, "a system built for virtualization." And they're upcoming only for the Intel stuff. IBM's POWER (architecture of its Unix machines) has had it since about 2003; IBM's Z (360/370/390) probably has had some of these since the early 90s, when it started integrating virtualization with its processor products.

The point of virtualization

Posted Oct 12, 2007 22:49 UTC (Fri) by roelofs (subscriber, #2599) [Link]

I presume you meant virtual to physical...

Yes, of course. Sorry for the typo!

The virtual page table is not involved in page translation. So it's the same speed with virtualization as without.

I don't understand why you think that. The guest app triggers a TLB (cache) miss, which means the guest kernel has to do a virtual-to-"physical" lookup in the page table, which involves one memory access for each level of the table. But its "physical" addresses are still virtual in the hypervisor kernel's space, which means you have to do a second virtual-to-physical lookup in the shadow page table, at a cost of three or four more memory accesses (for x86 and x86-64, respectively).

Those are probably included in my phrasing, "a system built for virtualization." And they're upcoming only for the Intel stuff.

Right--Ulrich explicitly stated that he was principally concerned with commodity x86-family hardware (and, to a lesser extent, Linux), and that certainly goes for me as well. I wouldn't be terribly surprised if IBM had some of this stuff back in the 1970s...

Greg

The point of virtualization

Posted Oct 12, 2007 23:02 UTC (Fri) by roelofs (subscriber, #2599) [Link]

Those are probably included in my phrasing, "a system built for virtualization." And they're upcoming only for the Intel stuff.

OK, maybe I didn't go back far enough for context. I think I now see what you meant by "a system built for virtualization"--you were implicitly excluding the extra set of memory accesses due to shadow page tables in software-based virtualization. But I misread that since you were replying to the author's comment about existing Dell hardware--which explicitly does not include IBM mainframe technologies. And your original statement was a flat denial of any performance penalty due to virtualization ("The virtualization layer doesn't slow the machine down, anyway."). It was more that part to which I was replying.

Greg

The point of virtualization

Posted Oct 13, 2007 2:31 UTC (Sat) by giraffedata (subscriber, #1954) [Link]

I agree the answer to the question, "does virtualization slow systems down" is different if one means virtualization per se than if one means virtualization in practice.

In the original comment, I thought I saw questioning of the very concept of virtualization -- i.e. why is everyone rushing toward this thing if it costs so much speed? So in addition to pointing out what I think the benefits are supposed to be, I also opined that it doesn't cost speed. I think the virtualization that people are presenting as the next big thing is virtualization done right, not necessarily what is practical today. E.g. companies are introducing new products specifically to enable virtualization.

The point of virtualization

Posted Oct 13, 2007 2:16 UTC (Sat) by giraffedata (subscriber, #1954) [Link]

The guest app triggers a TLB (cache) miss, which means the guest kernel has to do a virtual-to-"physical" lookup in the page table,

How would the guest kernel get involved? Page translation is done in the CPU. The guest app does a load instruction to an address he hasn't accessed in a while, so its translation is not in the TLB, so the CPU walks the shadow page tables to find to which physical page it refers, then loads the bytes into the register from there, then proceeds to the next instruction. The CPU doesn't even know the guest page table exists; the shadow page table maps the guest ap's virtual addresses to physical memory.

You seem to be saying TLB miss, but thinking page fault. When there's a page fault (the shadow page table does not contain a translation for the address in question), there is that multi-level lookup by the guest kernel and hypervisor, to ultimately add a translation to the shadow page tables. On systems that have guest page tables, that is.

The point of virtualization

Posted Oct 14, 2007 16:58 UTC (Sun) by roelofs (subscriber, #2599) [Link]

Yes, I think you've nailed it. I seem to have conflated the two issues.

Greg