MinGW and why Linux users should care

Richard: For some time I have been making Windows builds of libvirt available and, frankly, it was a real chore. I needed a Windows virtual machine to do it. But Windows is so frustrating to use and maintain: it doesn't come with any of the tools such as shells or version control that we are used to, and because I was only doing builds once a month or so I'd go back to it and find something had gone wrong that would require maintenance or even reinstallation.

During this time, we didn't routinely build libvirt for Windows. New code would inevitably break something. I had to fix things on Windows, then copy the code back to Linux and check that my fixes didn't break the Linux build, then come up with a patch, and all of this was complicated by the fundamental incompatibility of Windows with the rest of the world -- even simply copying code back and forth is irritatingly difficult when one machine is a Windows machine. (There's no ssh or scp or tar, files get executable bits set or have CRLF line endings, etc.)

At the same time we were getting a strong demand for the rest of our virt tools on Windows. Enough was enough. We decided that the only way to deal with this was to remove Windows from the equation. We wanted to build and test libvirt and the virt tools for Windows routinely (daily or more often), from the Fedora host, using the normal development environment. The way to do this is through cross-compilation (the Fedora MinGW project) and testing under emulation (Wine).

Debian & Ubuntu have been shipping the MinGW cross-compiler for quite a while, but it's important to say that the cross-compiler itself is the easy bit. The hard part about this project are the 50+ libraries and development tools that we ship and maintain alongside. Without those, just having the cross-compiler is fairly useless.

Dan: The libvirt project started a few years ago to provide an API for managing Xen virtualization hosts. Initially it was just a locally accessed C library, but over time the project expanded in scope to allow remote RPC access to the management APIs, and over other virtualization technology like QEMU, KVM, OpenVZ, LXC (native Linux containers) & User-Mode Linux. Shortly after we added support for RPC, a number of community members expressed an interest in using the client side from the Windows platform to manage their Unix hosts. Periodically people would contribute patches to make libvirt build on Windows, but soon after they were applied, new unrelated work would break the Windows build again.

It became clear that if the libvirt community was to officially support building a Windows client, then all developers needed to be able to easily test builds for Windows. The obvious stumbling block here is that most of our community developers do not use or even own Windows machines for testing. The MinGW project provides a cross compiler toolchain and stubs for the Win32 APIs to allow building of Windows executables and DLLs from a Linux host. Add in WINE and you can also run your cross-compiled build. MinGW and WINE are completely open source, so we can provide a very good level of support without ever having to purchase a Windows license or leave our primary Linux development environment.

We are not the first people to see the value in MinGW for supporting Windows platforms in open source software. Prior to the the start of the Fedora MinGW effort, Fedora developers would have to build all the cross compilers & libraries themselves. This is not particularly hard, but it is a lot of wasted effort to have everyone duplicating the work. Providing the MinGW compiler toolchain, and important libraries such as libxml, gnutls, libpng, libjpeg, GLib, GTK, etc directly in the Fedora repositories enables developers to focus on their own code, rather than the cross-compilers.

Richard: Dan and I work for a Red Hat group responsible for fostering the development of new tools and technologies. We have an eye to productisation and I spend quite a lot of time going to customer conferences and asking them what they want to see, but as for whether MinGW will make it into some future supported Red Hat product I cannot say.

Dan: Red Hat initiated development on the libvirt project and supports its ongoing evolution with significant developer resources. Red Hat wants the libvirt project to be the de facto standard for managing virtualization hosts, and the project community members want Windows to be a supported client platform. The work we are doing on the MinGW project in Fedora is thus a response to demand from the libvirt community for better Windows support in our releases. It is just a small part of our day job, alongside major libvirt feature development for Linux systems and in particular KVM & Xen.

Richard: Red Hat certainly cares about libvirt, and making libvirt available on the widest range of platforms. The alternatives to libvirt are interfaces like XenAPI and VMWare's APIs, which lock customers into proprietary technologies. Any way we can make it easier to provide open APIs and open source software even on closed platforms like Windows is a win for Red Hat, the Linux community, and even for Windows users.

Dan: As Richard says, this effort isn't about any particular Red Hat product. It is a community focused effort to address demand from libvirt users for better Windows client support. People are interested in open source virtualization technology like Xen and KVM, as an alternative to closed source solutions. Open source exists in a heterogeneous world though, and even if someone decides to migrate their servers to virtual machines on a Linux KVM host, they may still need to manage these servers from a Windows desktop. The MinGW project helps us maintain a reliable client build for the Windows platform, and thus lets a broader spectrum of users take advantage of open source virtualization technology. Growing the size of the libvirt community, and encouraging use of virtualization is what is important to Red Hat, and the MinGW project is one small part of that effort.

Richard: There's been some opposition, along the lines of "why are we helping Windows?". IMHO people who say that are ignoring both history and reality. First the history bit: the GNU project started off as a set of better compilers and command-line tools for the proprietary Unix systems of the day. I remember before Linux was around that you'd get some horrible system like HP-UX or (in my case) OS-9, and the first thing you would do would be to install all the GNU tools. Without real GNU grep, make, awk, bash, those systems were less than useful. Eventually when GNU got a kernel (Linux) we moved over to that system because it came with all the good tools.

Second the reality bit: Windows users are locked into proprietary applications and file formats, everything from Photoshop to QuickBooks to MSN to Illustrator. No Windows user can switch without first switching all their applications, which is going to be a very long transition process. Therefore we need a way to enable the developers of Gimp, GnuCash, Pidgin, Inkscape (to pick four out of hundreds) to easily build and test their software for Windows, so they can ship their software for Windows, respond easily to bug reports, and break that proprietary lock-in. Fedora MinGW does this - in fact we already used our compiler and huge chain of libraries to port Inkscape. [PULL QUOTE: Another thing we've found in porting to other platforms, is that it can generally improve the quality of the codebase. Different compilers and runtime environments expose different bugs in an application. The more combinations you can regularly build & test on, the better the overall quality of your code. END QUOTE]

Dan: The libvirt project started off with a strong Linux focus due to our immediate needs for a management API for Xen in Fedora and later RHEL-5. Over time the community has contributed patches to improve our portability to non-Linux platforms, in particular Solaris and more recently Windows. While Red Hat's focus is on Linux, enabling portability to other platforms is important because it grows the size of your developer community. Every significant open source project has a huge wishlist of features and nowhere near enough developers and testers to address them all. Cross-platform portability enlarges the pool of potential contributors. They may initially only send minor patches to fix portability bugs for Windows, but over time they can end up working on major new features that benefit every platform.

Another thing we've found in porting to other platforms, is that it can generally improve the quality of the codebase. Different compilers and runtime environments expose different bugs in an application. The more combinations you can regularly build & test on, the better the overall quality of your code.

Richard: My pet list would be:

• Don't write your own build system. Use autoconf/automake/libtool or cmake. That's not to say I'm a great fan of autoconf, but these really do make cross-compilation almost trivial.

  Autoconf-based programs can generally be cross-compiled by doing:

       yum install mingw32-*
       ./configure --host=i686-pc-mingw32
       make
  

• Don't try to run executables during the build phase. It doesn't work when you're cross-compiling.
• Do use pkg-config. And if you can't use pkg-config, then make sure your *-config program is a shell script, not a binary.
• Do use common, portable libraries such as glib, gtk, libvirt or any of our other libraries.
• Please use Fedora MinGW to routinely cross-compile your own code for Windows.

Dan: I have been pleasantly surprised at just how easy it has been to build many open source libraries with MinGW. Despite almost universal dislike for autotools, the applications which use autotools have been some of the easiest to port, particularly when it comes to building DLLs. The apps with home-brewed build systems have been much more involved. I definitely echo Richard's suggestion to stick to a broadly supported build system like autotools or cmake.

Any project which is serious about enabling support for Windows in their releases should make sure they are running regular automated builds & tests of their codebase. This is actually just good sense for any software engineering project regardless of whether Windows support is desired - it just happens to be particularly useful for configurations that developers rarely test on a day-to-day basis to avoid otherwise unnoticed regressions.

If you are not using a support library like GLib, QT or NSPR (which provides a degree of cross-platform portability) then seriously consider making use of Gnulib. This is a library of code which you can drop into an application, fixing POSIX API portability problems on various platforms. As an example, it replaces Winsock's socket() call so it returns real file descriptors that you can use in both read() and recvfrom(). It can't fix all problems - such as the lack of fork()/exec() on Windows - but if your application / library is written against POSIX, using Gnulib will significantly improve your portability across all Linux, UNIX and Windows platforms.

Richard: Scaling the project is a big challenge. Red Hat dedicates quite limited resources to this project. The only way we can scale it is if the application developers themselves start to use our tools to build and maintain their own programs. I would like to see everyone who has an important Linux app or library start building and shipping for Windows routinely. Bringing open APIs, apps and file formats to Windows users is important: It's important to Windows users because it breaks their lock-in and makes switching to a fully free platform easier down the road. It's important for you, because your potential audience of users will increase by a factor of 10x or 20x.

Dan: Spreading the package maintenance job across a larger number of Fedora members is an important task. There is a limit to how many packages a single person can do a good job at maintaining. To make it manageable we track & pull patches from the native builds to the MinGW cross-compiled builds of common packages. Ultimately we still need more package maintainers to look after the cross-compiled builds.

There are some core pieces of the open source ecosystem which do not work / are not fully portable to a Win32 environment. The most obvious one being DBus, which is used by an ever increasing number of apps for local RPC. There have been a number of efforts to port DBus, but none ever completely finished & merged into the official releases.

Richard: Get involved.

Dan: Cross platform portability is often beneficial to your project even if you personally only care about its use in Linux. In the libvirt case it is opening up use of libvirt & virtualization to a set of users who have only ever had access to closed source virtualization technology. Portability broadens the pool of potential contributors to your project. Open source developers on the various BSDs, OpenSolaris, and Windows all have the potential to make valuable contributions to your project.