Matt Domsch on Linux support at Dell

While I had seen a few PCs running Linux back in 1993 and 1994 while I was at MIT, it wasn't until I started at Dell in 1994 that I had access to PCs of my own that I could use for Linux work. I immediately installed Slackware on my "Engineering Workstation", a 486DX2-class system, and rebuilt kernels regularly for about a year. I even used Linux as a network stress-test tool for a while. I was a prototypical early Linux user, not yet a developer.

In 1999, Dell was receiving a lot of customer requests for Linux to be pre-installed on our servers. These requests came through our Custom Factory Integration team, who could install OS images, but who weren't set up to support Linux as a product. Dell saw the volume of these requests and decided to make Linux a formal product offering on our PowerEdge servers. I was just finishing a Master's degree at Vanderbilt and working for Dell when I was asked to help with "writing and fixing some device drivers". Little did I know it would become my life for the next 8 years as lead engineer then architect for the team.

Shortly thereafter, because sales were really taking off, we made Linux a "Tier 1" operating system on PowerEdge servers. Tier 1 means we won't ship a server to anyone until Linux, in particular our Linux partner of choice then, Red Hat Linux, ran well on the system. Previously, only Microsoft Windows and Netware had this distinction. In this change, we moved from a relatively small number of people developing and testing on Linux, to hundreds of engineers, software developers, and testers across the company working together to deliver stable products. The test organizations, OpenManage systems management software teams, groups that interact with our partner suppliers such as Intel, Adaptec, LSI, Broadcom, and the like, all stepped up to make sure their components were well supported.

We sold Red Hat Linux, starting with version 6.1 all the way through 9, then Red Hat Enterprise Linux 2.1 to present, on our PowerEdge servers. Along the way we added Precision Workstations to the mix. In 2004 we added Novell/SuSE Linux Enterprise Server on the servers, and in 2007, because of the strong demand seen on Dell IdeaStorm, we added Ubuntu on select desktop and notebook systems. In these 8 years, I've helped build a strong team of engineers who deliver these products, cycle after cycle.

I've now progressed to be a Linux Technology Strategist in Dell's Office of the CTO, helping to set Dell's future directions with Linux. In addition, I'm active in the Fedora Project a member of the Fedora Project Board.

Release schedules, and particularly, the dichotomy between new hardware release schedules, and schedules for 3rd party Independent Software Vendors (ISVs).

With Red Hat Linux, we were churning every 6 months. That works great for new hardware - there were lots of opportunities get new systems supported in the next release; but that churn is way too fast for ISVs. What we found is that if people were only using software that was included in the distribution, they didn't mind the frequent upgrades. But as soon as they had a dependency on a 3rd party application, the upgrade schedule just couldn't work, so they'd run older, less-supported versions of their distributions for longer. Companies like Red Hat and Novell/SuSE figured this out, which is why they've extended the time between major version releases, and extended each version's supported life cycle.

So now we've got longer stretches of time between OS releases, but the hardware still comes out on its own quicker schedule. We created DKMS - Dynamic Kernel Module Support, as our tool to help fill this gap. DKMS lets us update individual kernel modules (device drivers), without replacing the whole kernel. In the Dell factories, this lets us keep the widely tested "Gold" kernel for a given OS version, and replace just the specific kernel modules we must to fix major bugs and enable newer hardware. We also ensure that these fixes are rolled into the next Service Pack or scheduled Update from the OS vendor.

Dell expects our partner hardware vendors to provide fully open source / free software drivers, and to maintain them in the appropriate upstream project, kernel.org or x.org. This has been our policy since we started shipping Linux on servers in 1999, and continues to be our policy today. While we have a few holdouts, I praise the work Intel has done for their wireless and video drivers, and have high hopes for AMD/ATI video drivers given their recent announcements that they are working with the community to develop and maintain fully open source drivers.

This really comes back to the timing of releases again. When we started working with Ubuntu/Canonical this Spring, the Ubuntu 7.04 release was in beta, and already supported most of the hardware we needed for the initial product offering. But we knew we had newer platforms coming down the pipe that we wanted to pre-install Ubuntu on, and 7.04 didn't have drivers capable of working on that new hardware yet. DKMS helps, but if you're missing a storage driver, a network driver, and a video driver from the installation CD, it's still hard for a user to get an OS installed on that system. We produced the remastered Ubuntu CDs exactly to make it easy for users to install Ubuntu 7.04 on these newer systems.

We try our best to get future hardware supported in upstream kernel.org and x.org, and therefore in the distributions, as early as possible. The drivers we missed in Ubuntu 7.04 were included in upstream in time to be ready for Ubuntu 7.10's kernel freeze. Depending on the distro's release schedule, we may have to have code in upstream a year ahead of actual hardware availability to customers, and that's really difficult when the hardware itself churns faster than this. In parallel with the upstream submissions, we work with the distro teams to get those same drivers added.

Dell has hundreds of engineers and technicians around the globe who develop software for Linux, and test Linux our systems. We develop Linux for PowerEdge servers, Precision workstations, and select desktop and notebook systems.

The primary Linux Engineering team has two centers, one in Austin, Texas, USA and one in Bangalore, India, but it is one cohesive team. We run Linux on our own systems for day-to-day use. In addition to the Engineering team, Dell's Enterprise Test organizations put considerable effort into ensuring Linux "just works" on our systems. Our Technical Support organization has special teams of people with significant system administration and debugging experience, to handle customer issues. Many of these people have Linux certifications such as Red Hat Certified Engineer.

In addition to driver development and testing, we develop open source projects such as DRU (Disc Remastering Utility for Ubuntu), libsmbios, firmware-tools, biosdevname, and DKMS. These are available on linux.dell.com.

Noted above, DKMS is our tool that lets us provide an updated device driver without replacing the entire kernel. We use it as a stopgap mechanism, and work with our OS distribution partners to get those updates included into future distro-provided kernels.

DKMS leverages the kernel's own Kbuild system for compiling drivers from source code. It can handle lots of drivers, multiple versions of each, built for multiple architectures. It has a mode, the 'autoinstaller', where it can recompile driver source code on an end user's system when they upgrade their kernel. DKMS recognizes when the kernel's copy of a device driver is the same or newer than one provided elsewhere, so it can stop using the out-of-tree driver as soon as the in-tree driver is "good enough".

DKMS is extremely handy for testing changes to device drivers as you prepare it for inclusion in kernel.org, and for out-of-tree drivers who have not yet completed their task of being included in kernel.org.

DKMS is widely used in Mandriva and PCLinuxOS, and included in Fedora and Ubuntu. DKMS has been quite stable for while, accomplishing its primary goal. Recently we've enhanced it to add Ubuntu support - it can now create deb packages out of device drivers just as it makes RPMs and Red Hat / Novell Kernel Module Packages (KMPs). It can make driver disks for all of these distros too.

We also recently added a way to make it easy to download new driver RPMs specifically for your hardware. With a little more work, you will be able to "yum install" updated drivers when they're released by Dell.

As for the future, I see DKMS growing features to help driver developers, such as building driver disks for more distributions, and building better, more easily consumable KMPs. I'd like to see it included in more distributions, but I don't expect any huge changes - it does its job well.

While our sales of desktops and notebooks with Ubuntu pre-loaded have met our expectations, I personally hoped that Ubuntu sales would be through the roof. In August we expanded sales from US-only into Germany, France, and the UK, so we're just starting to see the sales there. We're working to add the next version of Ubuntu, 7.10, on the existing set of notebooks and desktops. What additional systems get offered depends on sales.

As for the number of distros supported, Dell already sells Red Hat Enterprise Linux on our PowerEdge servers and Precision workstations, Novell/SuSE Linux Enterprise Server on our PowerEdge servers, Ubuntu on our desktops and notebooks, and we've announced plans to add Novell/SuSE Linux Enterprise Desktop on notebooks and desktops in China. That's a pretty wide coverage already.

One challenge we'll continue to have is the sheer number of Linux distributions available that we could offer on our systems. Clearly we won't validate and sell all of them. Our method for combating the proliferation of distributions is to work with the upstream projects to get our hardware supported, and then to let each distribution pull from those upstream projects. This policy let us add Ubuntu to our product line in record time - it would have taken much longer had we needed to write and validate all the drivers for those systems from scratch. This policy also lets users choose which distro they run based on their business needs or personal preferences, not solely on what Dell chooses to validate.

A second strategy we're pursuing which should help is virtualization. If instead of testing each system/distribution combination, we could test a smaller number of system/hypervisor combinations, and the distros themselves could test their own hypervisor/distro combinations, then we get the best of both worlds - end user choice of distributions, with the same or lower development effort needed by each hardware manufacturer.

We're also going to continue looking for ways to make it easier, faster, and less expensive to bring new products to market. If we had waited to train phone personnel in all our offices around the globe, that would have been slow and expensive. With Ubuntu we've seen that online support offerings work well for most users, and that they don't want to pay for telephone-based support they don't feel they need. For us, online support methods are quicker and easier to develop, and they scale to many more users far better too. By delivering an appropriate level of support for these users, everyone saves time and money. Our public mailing lists, forums, and web pages at linux.dell.com do exactly this. And for those who want paid-for telephone support, we offer that as well.

One last plug. If you are buying a sufficient number of systems, Dell can do anything you need. Your custom-crafted Linux OS installed on a system to make a hardware appliance - no problem. Custom-crafted servers to fill your data centers - no problem. You won't find these offered in the Sunday newspaper, but we can do it.