Two FOSDEM talks on Samba 4

Much as some of us would love never to have to deal with Windows, it exists. It wants to authenticate its users and share resources like files and printers over the network. Although many enterprises use Microsoft tools to do this, there is a free alternative, in the form of Samba. While Samba 3 has been happily providing authentication along with file and print sharing to Windows clients for many years, the Microsoft world has been slowly moving toward Active Directory (AD). Meanwhile, Samba 4, which adds a free reimplementation of AD on Linux, has been increasingly ready for deployment. Three short talks at FOSDEM 2018 provided three different views of Samba 4, also known as Samba-AD, and left behind a pretty clear picture that Samba 4 is truly ready for use. I will cover the first two talks in this article, and the third in a later one.

War stories

The first talk, "SaMBa-AD, it works", was given by Denis Cardon from the French system integrator Tranquil IT. He specializes in deploying Samba 4, and was keen to tell of his field experiences in doing so. Samba has been big in France for some time, which Cardon attributes variously to the "free-as-in-beer syndrome" (because there are no per-seat client license costs with Samba), the "free-as-in-speech syndrome" (a general love of liberty), and "the General de Gaulle syndrome" (a dislike of American products being forced down French throats).

Samba 3 (which he referred to as "Samba-NT") has allowed some sites to stay clear of AD for much longer than a Microsoft NT-based domain controller infrastructure would have permitted; LDAP, SMB 2 and SMB 3 can all be used on a Samba 3 domain. But it's getting increasingly difficult to keep those sites running with newer client operating systems on the desktop; Windows 10 clients, for example, can only authenticate if you force the use of older protocols on the domain controller (DC). So over the past few years much of Cardon's business has been upgrading sites from Samba 3 to Samba 4. Of late, however, more and more sites are paying his company to come and migrate existing Microsoft AD deployments to Samba 4; word is spreading that Samba 4 is ready for production.

He has deployed Samba (both 3 and 4) in the French central government, in regional and city administrations, the French defense sector, schools, and universities. His clients include Samba 4 sites with five thousand desktops and ten thousand users, and sites with dozens of domain controllers; one site has 80. One client has asked about a 120,000 user deployment, but apparently Andrew Tridgell Bartlett, when asked, advised waiting for Samba 4.9, due out later in 2018.

Much of the rest of Cardon's talk was made up of amusing war stories: the site where the only working domain controller was a thirteen-year old NT 4 system the interior spaces of which, when opened, were entirely filled with dust. Deployments that had to be hastily finished before hurricanes took the power down, and those held up by coups d'état (though in one latter case, local representatives advised that the usual delay caused by a coup d'état was a week, so eight days later they reconvened and completed the migration). There was another migration where every system was successfully migrated except the computer that ran the door-entry-card system, so the following morning Cardon had to negotiate with security to break him into the site so he could fix it. Entertaining as these all were, the important lesson of the talk was that Samba 4 is ready to take on authentication duties for very large sites with complex requirements.

But Samba 4, as it ships with some modern server distributions, does not quite reflect this reality. CentOS 7, for example, released more than two years after Samba 4, does have a samba-dc package. But this package contains precisely one file, /usr/share/doc/samba-dc-4.6.2/README.dc, which explains that AD support is not available in Samba 4 as supplied by Red Hat. One can, of course, build Samba 4 from source, but this is not an approach favored by many production system administrators.

Vendor Samba

The second talk, "Samba AD in Fedora", given by the Red Hat Samba maintainer Andreas Schneider, explained the historical context behind the current state of vendor-provided Samba, and how things are changing. AD depends fundamentally on Kerberos, a network security protocol developed at MIT in the late 1980s. MIT Kerberos was for many years the reference implementation. Enterprise distributions, noted Schneider, are required to ship with MIT Kerberos if they want to do business with governments, and in particular with the US government. Everything in the distribution that is "Kerberized" will be integrated with MIT Kerberos. There is another implementation, Heimdal, but it is unwise to mix both implementations on a given system because the on-disk representation differs, and because there are symbol clashes if you try to load both libraries into a single application; the presence of one effectively excludes the other.

Samba, however, is built against Heimdal Kerberos. In response to a later question, Schneider said he did not ascribe this to US export restrictions on crypto. It was primarily because at the time the AD implementation was being developed, MIT Kerberos development was moribund. Heimdal, meanwhile, had an active community, which was open to implementing features that Samba needed. Volker Lendecke, who also spoke on Samba, noted from the audience that a similar situation had happened with LDAP: Samba needed an implementation, but the OpenLDAP community was resistant to accepting Samba-related patches, so Samba developed its own LDAP server. Thus, Samba went with Heimdal Kerberos. Meanwhile, Heimdal development had slowed dramatically (though it is now recovering), possibly because the lead developer went to Apple. MIT Kerberos development, however, has had a resurgence.

Those changes made for an auspicious time to try to make Samba work with MIT Kerberos. The first step, said Schneider, was to develop a test harness. Samba has an extensive test suite; Schneider noted some 15,000 tests in 2,000 test suites that take him about three-and-a-half hours to run. If code refactored to work with MIT Kerberos couldn't pass those tests, it wasn't going to get into upstream.

So, in February 2013, he started writing cwrap, "a set of tools to create a fully isolated network environment to test client/server components on a single host, complete with synthetic account information, hostname resolution, and privilege separation support". Or as he then put it more pithily, an equivalent of the Matrix for applications: a way to run applications while telling them lies about the environment in which they are running.

A year later, the team was ready to run Samba with cwrap, and started working on the refactoring. The developers also started finding, reporting, and, in some cases, fixing bugs, both in Samba and in MIT Kerberos. By April 2015, the patch set to Samba had grown to 140 patches. Rebasing had become a painful daily activity, not least because the team's branch of the code was generally ignored by other Samba developers, so the team had to keep up with everyone else's development as well. Most of the test suites were passing at this time, but 69 were still failing. Nevertheless, they started to push their patches upstream.

In April 2016, work was stalled for four months by the discovery of the Badlock bug. Just as that was fixed, MIT Kerberos 1.15 was released, and Schneider and colleagues discovered that the developers had removed the API function which allowed memory management for KDB modules, thus giving their refactoring a nice big memory leak. Hasty conversations with the developers about putting it back ensued.

By January 2017 they were down to a single test failing, and a couple of extant issues. These were fixed and, on 30 April 2017, the code hit the Samba Git master branch. Five months later, Samba 4.7.0, the first version with official MIT Kerberos support, was released. Work on the correct way to package this ensued, and on 14 November 2017 Fedora 27 was released—with Samba AD. No particular timeline was given for the move into RHEL, but that is how technology generally percolates through the Red Hat environment. Should you wish to run the Fedora 27 version, Schneider said, all the important stuff works (single domain controller, forest, and external trust relationships). Installation and provisioning is now just three lines:

    # dnf install samba-dc
    # samba-tool domain provision
    # systemctl start samba.service

More advice on running Samba followed. Samba requires DNS, because AD does. It has its own internal DNS server, but it also provides a module to work with BIND, called samba-dc-bind-dlz. Schneider feels that the module is a horrible hack that directly manipulates the AD databases and thus has write access to them. His strong advice is not to use it, and to use the Samba internal DNS server instead.

AD administration in Samba is still fairly painful; Windows ADUC tools are probably the easiest way to do it. samba-tool is still fairly painful to use, and its error messages tend to consist of Python stack traces, but it is improving all the time. Lendecke was kind enough to demonstrate to me, there and then, the new "samba-tool user edit foo" functionality, which allows all of a user's information in AD to be seen and edited in one screen; new objects can be added and the tool complains properly if you try to add an object that is not in the schema. A limited but functional module for Cockpit has recently been written, which will at least allow you to provision and interrogate an AD domain. Patches are encouraged.

Some things are still not working. Audit logging is a work in progress, and should come in Samba 4.9. Smartcard support might still be missing; he thought it probably would work, but there are no tests for it, so he can't be certain ("we normally say at Samba ... untested code is broken code"). Read-only domain controllers are unsupported in MIT Kerberos and so in Samba, and quite a lot of work inside Kerberos would be needed to support that. That brought his talk to a conclusion, eliciting some well-deserved applause.

Schneider's talk can be seen in its entirety here and, for those who like system administration war stories, Cardon's talk can be seen here.

[We would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Brussels for FOSDEM.]