LWN.net Weekly Edition for February 15, 2018

Welcome to the LWN.net Weekly Edition for February 15, 2018

This edition contains the following feature content:

• DIY biology: an LCA talk about people taking biology research into their own hands — sometimes literally.
• A GPL-enforcement update: Karen Sandler updates the community on what is happening around enforcement of the GPL.
• A report from the Enigma conference: a report from a different sort of security conference.
• The rest of the 4.16 merge window: what was to be found in the last 5,000 changesets pulled into the mainline before 4.16-rc1.
• Shrinking the kernel with an axe: Nicolas Pitre continues his series on shrinking the kernel by looking at the things that can be cut out of it.
• Two FOSDEM talks on Samba 4: A summary of two discussions on the Samba Active Directory code in the real world.
• Authentication and authorization in Samba 4: a detailed technical talk on how Active Directory authentication works.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

DIY biology

A scientist with a rather unusual name, Meow-Ludo Meow-Meow, gave a talk at linux.conf.au 2018 about the current trends in "do it yourself" (DIY) biology or "biohacking". He is perhaps most famous for being prosecuted for implanting an Opal card RFID chip into his hand; the Opal card is used for public transportation fares in Sydney. He gave more details about his implant as well as describing some other biohacking projects in an engaging presentation.

Meow-Meow is a politician with the Australian Science Party, he said by way of introduction; he has run in the last two elections. He founded BioFoundry, which is "Australia's first open-access molecular biology lab"; there are now two such labs in the country. He is also speaks frequently as "an emerging technology evangelist" for biology as well as other topics.

Background

Biohacking was started in response to the belief that academia is slow and expensive. The knowledge and thinking is "siloed"; people on the same floor of an academic facility may not know that they are working on the same problems. It excludes people who have an interest but lack the higher degree that is considered required.

So biohacking came about because non-professionals wanted to participate in molecular biology, he said, but didn't want to do "citizen science". Going out and collecting data points for "real" scientists so they can publish papers "is boring". The biohacker movement wants to turn citizen science into "civic science".

The civic science term came about during the Fukushima disaster when MIT Media Lab director Joichi Ito ("who also doesn't have a degree, by the way") was trying to figure out how to use hacking and hackers (in the original sense of the term) to help out. There are things that hackers do well, such as moving quickly and thinking outside of the box, that might be applied to the problems faced by the disaster victims.

Out of that came Safecast, which was an important moment for Meow-Meow because Safecast made the transition from citizen to civic science for him. The Japanese government ran out of Geiger counters and other devices to measure the radiation levels from the disaster, so Safecast worked with hackers to build their own—reducing the cost from $6000 to $600. These devices were then used to measure radiation levels throughout the region. This was civic science because the users were doing more than just collecting data points; they were also interpreting and processing the data and making decisions based on it. The maps generated by the project were much better than any others, to the point that the government adopted them after three months.

Having all of this open data led to innovation, he said. He showed a video of a biohacker from Fukushima who linked the color of some lights in the wheel of his bicycle to one of the Geiger counters. So as he traveled around at night, he could get an idea of the radiation level of the area.

Biohacking consists of microbiology, molecular biology, building your own equipment, bioinformatics, and grinding, which is "the act of putting technology inside your body". Some of that might be done in an academic setting, but in general it is not. Biohacking takes a more "tinkering kind of approach", rather than "hypothesis-driven" as academia tends to be.

He founded BioFoundry three or more years ago and "it has been a wild journey". The first BioFoundry lab was in a small space in an artist coop near the University of Technology Sydney (UTS) campus where LCA was held, Meow-Meow said. He and some folks that he had been meeting with for several years gathered equipment out of dumpsters and built some other equipment for the 16m² lab. The space was pretty dodgy and they outgrew it quickly, but "the cool thing is we were doing some really fun science".

Two iterations later, the lab is larger and better equipped. The lab has Physical Containment 1 certification, which allows researchers to work with viruses and "hack bacteria". The certification "enables us to do most of the cool things except genetic modification on plants", he said.

Biohacking projects

The key project being worked on currently at BioFoundry is Open Insulin, which was started by another biohacking group in Oakland, CA. Biohacking is generally aimed at making the world or people better, he said; Open Insulin is about "making open-source insulin" using processes that are not covered by patents. Insulin used to be made in pig pancreases, but is now mostly made by bacteria that have been combined with a human gene. Prices of insulin are going up and production of it is going down, to the point that people in third-world countries and even China do not have access to it, which is "criminal", Meow-Meow said.

He then took attendees on a bit of whirlwind tour through some of the projects that are being done in the biohacking world. David Ishee is a dog breeder in the US who has become "one of the best biohackers in the world". He wants to fix genetic errors in dog breeds, many of which were introduced by humans using selective breeding. Ishee has run into problems with the US Food and Drug Administration (FDA) trying to classify what he is doing as creating drugs, "which is completely bizarre". At the time of the talk, the US government was (briefly) shut down, which made Ishee "very excited", Meow-Meow said, to a round of laughter.

Sascha Karberg is a biohacker who had a problem with waking up each morning to fresh dog poop on his lawn. "Instead of doing what any normal person would do, he decided to overengineer the solution to the problem". He took tennis balls to the local dog park and threw them for each dog, collected the saliva from them, and did basic DNA sequencing on each at a local biohacking space. After sequencing the poop, he was able to find out which dog was making the mess, confront the owner, and stop the poop problem. It is an example, Meow-Meow said with a chuckle, of how biohacking can be used to solve a problem.

Another part of the biohacking movement is the creation of equipment to be used as part of the experiments or for data collection. People who can write code, do electronics, or run a laser cutter can contribute to efforts at local biohacker spaces to build this equipment. Much of it is "upcycled computer gear"; instead of a basic setup costing $20-30,000, it can "be built for like 500 bucks", he said.

Another way that people without a biology background can help is in providing ways for visualizing the data. Some of his favorite projects are ones where people from different skill sets get together to see what problems they can solve. One of those is a river watch project [Google Translate link] by an Indonesian group called Lifepatch [Google Translate]. The group made simple coliform bacteria detectors that were used to measure the amount of sewage reaching various waterways. That data is collected and displayed so that people can determine which parts of the rivers are safe to drink from. Each biohacker space in the world seems to focus on a different kind of problem, he said.

Another example is Josiah Zayner, who is "solving very different problems". He had a problem with his digestive tract, so he did a fecal transplant, which was so successful that he quit his job with NASA to do molecular biology full time. Zayner is perhaps most famous for his web site "The ODIN", where he sells reagents and kits to anyone as part of the "democratization of science" movement. He got in trouble with the EU for shipping CRISPR kits for bacteria to Germany; the EU claimed he was shipping pathogenic E. coli bacteria, but that turned out not to be the case. Zayner also has kits for use on humans ("though I wouldn't recommend that"); Zayner has used the kit on himself, however.

We are at the point where "lots and lots of people" are doing experiments using CRISPR, he said; it is "a brave new world". A "kind of sad story" is that of Tristan Roberts, who is infected with HIV. Roberts was frustrated about the advent of various experimental drugs for treating HIV that he did not have access to. So he recently injected himself with a therapy that looks promising and stopped taking his anti-HIV medication (Meow-Meow said that discontinuing the medicine was not popular in biohacking circles). It is too early to know the results, but there are a number of ethical questions that arise: should Roberts be allowed to do that? What if people in a country with socialized medicine start treating themselves that way; should the medical system be required to cover that?

Opal implant

Meow-Meow then moved on to his experiments on himself. The Opal card, like other RFID cards, consists of a chip and an antenna. The antenna is quite long in the card because it goes around the edge of the card many times. RFID works by the terminal providing an RF field that causes power to be generated by induction in the antenna that then powers the chip; the terminal and the chip can also communicate via the RF. When he first started talking with folks about implanting the Opal chip, it was the antenna that everyone thought would be the biggest problem. He actually has three different implants at this point (two for Opal, one other). One Opal implant is in his thumb, but doesn't work well because the antenna is not the right shape to interact with the Opal terminals.

So getting the right antenna was key, but the designer of the antenna, Amal Graafstra, lives in Seattle, which made it somewhat time-consuming to do. Graafstra started trying out different kinds of antennas, some of which ended up in some his own implant projects. Graafstra eventually settled on a small (roughly 3x1cm) printed circuit board with the antenna looping at one end; the Opal chip was dissolved out of the card with acetone and soldered to the board. The board was laminated with a bio-compatible plastic coating and was implanted into the side of Meow-Meow's hand on the side opposite his thumb.

Meow-Meow showed video of using the implant in his hand at an Opal terminal. After that video appeared on television, "I got in trouble", he said. The Sydney transit authority wanted to cancel the chip in his hand, which made him leery of actually using it. When the Opal system was first put in place, various folks asked why the transit authority needed to collect personal information as part of the sign-up process. The agency said "you can trust us, we'll never misuse your data", but "the first fucking chance they got to misuse my data, they did" by shutting down a different Opal card that he had in his name—on the day he did the TV story.

He said that he personally does not trust the government with his metadata "as far as I can throw them". But this incident does give him a way to talk about metadata retention as part of his campaign in the next election cycle. It is a good demonstration of why governments should not be allowed to retain metadata. Due to his offense (having an invalid ticket, which carries a $200 fine), he has to go to court on March 16. It will be the "first cyborg law case in Australia", he said with a grin; "it will be a bit exciting".

The other implant he has is a "little glass bead" in his hand. It is similar to the identification chips that some dogs and cats have and was also designed by Graafstra. In the future, Graafstra will be selling the VivoKey, which has roughly the same form factor as Meow-Meow's Opal implant, but will do two-factor authentication. The VivoKey has a megabyte of space, "which is huge"; the chip in his thumb has 868 bytes of storage by comparison.

He pointed to an article in ScienceDaily about the longevity of batteries in devices such as pacemakers (foreshadowing, a bit, Karen Sandler's keynote later in the week). The incentives for device makers are to reduce the warranties on battery life in order to make more money, he said, since "they can't make people more sick". The medical cost to replace a pacemaker battery in the US is $135,000, so he thinks we will see more efforts by patients and others to find alternatives. If someone gets an "off-brand" battery, though, will that violate the terms of service or warranty of their pacemaker? There will be many ethical and other questions along those lines in the near future, he thinks.

There are also efforts underway to implant more functional computers, with sensors, actuators, and wifi/Bluetooth communication, rather than the relatively dumb RFID devices. Beyond that are implants for decoration, like an LED implant that displays moving star patterns under the skin. Or using implanted magnets coupled with external sensors to simulate the echolocation ability of dolphins. And so on. The people behind these kinds of projects would seem to agree with the "Bioquisitive" slogan on Meow-Meow's t-shirt. He encouraged anyone so inclined to get involved with BioFoundry or their local biohacker space to "help us make cool science".

In the Q&A session, he noted that his Opal implants were not something he was going to keep permanently, but that they had opened up an interesting conversation. The Opal chips are proprietary, which is what the audience member was asking about but, he asked, why do they need to be? If he wins his court case, he thinks that will change how people are allowed to interact with this technology.

The advantages of an implant over the card or other mechanisms are numerous, he said. "It is 100 times cooler to implant it than to wear it". He can't lose the implant and he can swim with it. In addition, it is not obvious that he even has it, so people wouldn't even necessarily think to steal it (which would also be hard, of course).

There are some tricky ethics and safety questions surrounding these uncontrolled biohacker labs, he said. But the BioFoundry lab is subject to the same regulations as any other lab. BioFoundry works directly with the government and informs it of what the lab is doing. That may or may not be true for other labs, however.

A YouTube video of the talk is available.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Sydney for LCA.]

Comments (17 posted)

A GPL-enforcement update

While there is a lot of software distributed under the terms of the GNU General Public License, there is relatively little enforcement of the terms of that license and, it seems, even less discussion of enforcement in general. The organizers of linux.conf.au have never shied away from such topics, though, so Karen Sandler's enforcement update during the linux.conf.au 2018 Kernel Miniconf fit right in. The picture she painted includes a number of challenges for the GPL and the communities based on it, but there are some bright spots as well.

Sandler is the executive director of the Software Freedom Conservancy (SFC), which is occupied with a number of activities beyond license enforcement. SFC is the home of the Outreachy project, for example, and serves as an umbrella organization for a long list of free-software projects. The enforcement efforts were the focus of this discussion, though.

She started with an overview of the GPL and how it works. Due to the way the kernel project came about, many of the developers own the copyrights for at least some of the work they have contributed. The SFC represents a group of those developers that came together and asked for help in the enforcement of their copyrights. This work is always done with an eye toward bringing about compliance and helping the free-software ecosystem in general. SFC is a tiny charity with limited resources, so it has to be strategic with its actions. The work is "fascinating" and "so fun", though she acknowledged that being a lawyer may distort her view of the world a bit.

A number of exciting things have happened in the last year, including the release of the enforcement statement signed by a long list of kernel developers — and many of their employers. This statement was somewhat motivated by a specific kernel developer [she was referring to Patrick McHardy] who has been launching aggressive enforcement actions in a way that she described as "not very community-oriented". In reaction, the Linux Foundation's Technical Advisory Board came together to try to clarify how the community saw enforcement and to distinguish that vision from what McHardy was doing.

The statement codifies a practice that the SFC has used for years: applying the termination conditions from version 3 of the GPL to software licensed under version 2 only. The GPLv3 language makes it clear that violators can restore their license to the software by coming back into compliance with the terms of the license. This change "makes sense", she said; inadvertent violations happen all the time, and mistakes should not carry such severe consequences. With nicer terms, today's violator can more easily become tomorrow's contributor.

Right after that happened, four companies (Facebook, Google, IBM, and Red Hat) came out in favor of the new terms; they issued a statement applying those terms to all of their GPLv2-licensed software.

Another interesting event was the DJI Drones affair. Jon Sawyer tweeted that, after a couple of unsuccessful attempts to get DJI to release the source for the GPL-licensed software it ships, he had created and published a root exploit for the company's drones in retaliation. That brought a lot of attention to GPL compliance and just how frustrated developers get when their copyrights are not respected. This sort of action doesn't fit within the SFC's enforcement principles, but she recognized that others may see things differently.

Savvy violators and powerful vendors

Enforcement activities within the SFC are slowly chugging along. The SFC is encountering a lot of delay tactics by companies; the desire is to give companies the time they need to comply, but a lot of these companies are clearly just stringing things along. The effort to get VMware into compliance went on for over four years before the lawsuit was filed, for example. It is important to give companies time to look at the situation and come into compliance, but it seems that some have figured out that they can use that time to delay the whole process.

There has also been an increase in "savvy violators". It used to be that most violations were inadvertent — the companies involved simply didn't understand what their obligations were. Bringing these companies into compliance was a relatively easy task. Now, there are many companies that "know exactly what they are doing". They have concluded that, since there are so few lawsuits around GPL compliance, they can get away with ignoring the GPL. This has been a significant change over the last five years.

When asked about these savvy violators, Sandler expanded a bit. Many companies have had their lawyers do a "tight legal analysis" describing just what the company can get away with. That gives product managers a tool; they think that they can gain an edge by holding back as much as possible until the product reaches its end of life. Technologists within such companies should make more of a fuss internally about such practices, she said.

Another problem is the "powerful vendor". In many cases, violations originate with a particular vendor, and the companies that buy from that vendor are too worried about damaging the vendor relationship to push for compliance. The company marketing the final product looks like the violator, but it is really a victim of the upstream vendor's behavior. This company has no ability to fix the violation itself, and is afraid to work to get the upstream vendor to comply. The SFC has been trying to come up with creative solutions to this problem. One might be to get the downstream vendor to sign a statement to the effect that it has not received the necessary source from the upstream vendor; the SFC would then not pursue enforcement activities against that company.

The increasing interest in security is having a helpful effect instead. It has become relatively easy to convince corporate lawyers that their company needs to obtain the full source code from all vendors so that it will be in a position to deal with any security issues that may arise.

One other ongoing problem is proprietary kernel modules. The SFC has even seen companies patching EXPORT_SYMBOL_GPL() to get the kernel to accept their non-GPL modules, a development Sandler described as "fascinating, shocking, and deeply upsetting".

Copyright retention

The prepared part of the presentation ended with a shift to the topic of individual copyrights. It's a common pattern in our community that projects are started by individuals who naturally own the copyrights on the work they create. The Linux kernel started this way, and it led to an interesting license dynamic, since the individuals involved retained a great deal of influence over the direction of the project.

In a successful project like the kernel, the amount of work done by companies grows over time; this is a good thing. But it does lead to an increase in the amount of copyright in the code that is owned by companies. Much of the success found by the Linux kernel has to do with its origin in copyleft, which has enabled a "beautiful balancing act" with many people and companies collaborating to make the kernel better. As we shift to a situations where companies have more control, that balance can be lost.

Free software works well because the community is in control, but if we do not work to keep that control, we'll cede a lot of our power to companies. One of the great things about being a kernel developer is the ability to go from one company to another and do the same work, or something close to it. Developers are in control and they get to work on what they love; that is possible because the GPL is a real equalizer. Without compliance, we'll lose that benefit, and the kernel will become less special. We may not see the kind of collaboration we've had in the past.

To try to head that off, the SFC has been encouraging developers to negotiate to be able to keep their copyrights in the work they do. There seems to be an increase in the number of companies that are allowing retention of copyrights, she said; as the practice grows, it may become truly common as companies use it as a recruitment tool.

Strong copyleft is in the industry's best interest, she said, but companies driven by short-term interests often see things differently. That has made it hard to find places to have discussions about topics like copyleft and copyright retention. One of the things that makes LCA special, she concluded, is that it is still a space where we can talk about these issues.

Questions

A member of the audience asked about whether developers should bequeath their copyrights in their wills. That turns out to be hard to do in many jurisdictions; it may require an appraisal of the value of the copyrights and may cause the beneficiaries to have to pay taxes. A better way is to set up a trust that can hold and manage those copyrights; this can be done while the developer is still alive.

Another person asked: is there value in reaching out to universities? Sandler agreed that there is; we in the community used to spend a lot of time at universities, she said, but that practice has mostly stopped. It would be a good thing to resume. Projects like Outreachy are helpful in this regard.

The final question was asked by Matthew Wilcox, who noted that he worked for LinuxCare once upon a time and has no idea where the copyrights in his work have ended up. Is there any initiative out there pursuing the copyrights held by zombie companies? Some of those copyrights could probably be obtained cheaply. Sandler responded that the idea is interesting but would require some thought. The SFC puts a lot of effort into ensuring that its actions reflect the wishes of its community. Picking up these copyrights could provide a stronger voice in negotiations, but could also look a bit like a troll attack. It could give the appearance of acting in bad faith. On the other hand, acquiring those copyrights would keep them out of the hands of others who might try to exploit them; it is something requiring further thought.

[Your editor thanks the Linux Foundation and linux.conf.au for assisting with his travel to the event.]

Comments (10 posted)

A report from the Enigma conference

The 2018 USENIX Enigma conference was held for the third time in January. Among many interesting talks, three presentations dealing with human security behaviors stood out. This article covers the key messages of these talks, namely the finding that humans are social in their security behaviors: their decision to adopt a good security practice is hardly ever an isolated decision.

Security conferences tend to be dominated by security researchers demonstrating their latest exploits. The talks are attack-oriented, they keep a narrow focus, and usually they close with a dark outlook. The security industry has been doing security conferences like this for twenty years and seems to prefer this format. Yet, if you are tired of this style, the annual USENIX Enigma conference is a welcome change of pace. Most of the talks are defense-oriented, they have a horizon going far beyond technology alone, and they are generally focused on successful solutions.

The brief talks are only twenty minutes long with ten additional minutes of Q&A; the conference is clearly trying to emulate TED talks in terms of quality and focus. All speakers go through three separate training rounds where they present their talk to each other remotely to help them polish the talks. Gender and racial diversity plays a big role at Enigma and the conference appeared to have reached something close to gender parity this year; the fact that there were regular queues in front of the female washing rooms was a constant source of comments.

I was attracted to Enigma when I saw the video [YouTube] of Rob Joyce's talk from the first edition of the conference in 2016. Joyce was the director of US National Security Agency's (NSA) Tailored Access Operations department; his presentation was about how to protect against nation-state attackers like him. The talk impressed me with his forceful insisting on basic security practices. So I attended in 2017 and returned for this year's Enigma conference in Santa Clara.

Social influence

It was the research into user behavior that stuck with me this time. A perfect example of a high-quality Enigma talk on this topic was the presentation by Sauvik Das from Georgia Tech. He talked about his research into user behavior based on several user interface tests at Facebook and the adoption rate for various security practices. It is an important topic since the whole industry builds many security systems; companies introduce them into their services but the majority of users ignore them, which leaves their accounts and their data at risk.

First, Das tried to measure social influence on security behavior. He measured the adoption of three Facebook security features: (1) login notifications that inform a user when a login has been performed, perhaps with stolen credentials; (2) login approval, which is Facebook's flavor of two-factor authentication via mobile phone text messages; and (3) trusted contacts that can help with account recovery by identifying a user that has been locked out of their account.

He looked at data from 750,000 Facebook users who had adopted one of these features and 750,000 users who had not started to use any of them. Measuring the social influence was fairly simple: how big was the exposure level among Facebook friends for a given feature? Did 1% of friends use it? 5%? 10%? And so on. At each exposure level, he compared the adoption rate and came to the conclusion that there is a strong correlation and thus social influence. But there was also a surprising finding: early adopters can have a negative effect on the adoption rate of a security feature. If they come across as paranoid or nutty geeks, then their use of a feature will stigmatize it and hinder the further adoption among their friends. But this does not negate the overall conclusion: the more exposure users get to a feature via their friends, the more likely they are to adopt it.

After laying the base in such a way, Das moved on to a series of experiments where he tried to influence the adoption rate of 50,000 Facebook users by varying the exposure to social influence. He would present them with an announcement text introducing the security features. He varied the wording of the announcement text and looked into changes of the adoption rate. One group of people would get the plain announcement. Other groups were informed about the absolute number of friends using a feature; others got a percentage of friends.

There was a positive effect with all the announcements and any form of social information improved the adoption rate even if the increase was not dramatic. The click-through rate of the announcement grew by around 30% and the adoption rate by about 10% if the announcement cited the raw number of friends using a particular feature, which was the experiment that performed the best. 10% is not all that dramatic, but if you put it in perspective with the engineering costs of a feature and the relative simplicity of sharing a bit of information about the adoption rate among a user's peers, then the 10% of additional adoption based on a single announcement has a reasonable return on investment in terms of improved security.

Taking a step back, Das named some characteristics that help users with their security practices. Security systems need to be observable so that people can actually see and emulate the good security practices of their friends. In addition, security should allow people to act as stewards. This is based on the interesting observation that people do not care too much about their own security but they are very concerned with the security of their loved ones. Designing security systems that allow people to act on these concerns is therefore desirable.

Staff security

Das's presentation was part of a series of talks that covered security behavior at the conference. Another example was Masha Sedova's talk that looked into the ways companies can improve the practices of their staff. It would seem that awareness training sessions don't go very far and their effect is quickly lost. Otherwise, the last smoker would have quit by now. But the motivation to quit smoking — or use strong passwords — just is not there, no matter the awareness. Sedova had the numbers to back this up. She came to the conclusion that knowledge is not enough. It also takes motivation, the ability to adopt a better behavior, and then a trigger to actually start.

The harder it is to do something, the higher the motivation needs to be to change a behavior, start a new habit, etc. In practice this means that simplifying the steps to adopt a secure behavior is essential. The secure way to perform a task should always be the easiest way perform it. And adoption will definitely follow.

Where this is not possible, you need motivation. Yet motivation does not always come by itself. She named two typical situations that tend to grow motivation; one is a scheduled audit of a team or a project where people grow more and more motivated the closer the effective date gets. After the audit, the motivation disappears rapidly. The second situation is a security incident that immediately brings a big motivation to change the security practices. Yet this motivation does not stick, it slowly degrades as time goes on.

How can you influence this motivation as a company? It definitely pays to work with these events. Use them to your advantage when trying to introduce new practices. Sedova quickly ruled out money or punishments as an effective means of motivating your staff with few exceptions. For the majority of people, you need positive reinforcements as a way to create intrinsic motivation. Here she named public recognition of good performers, competition in company events, altruism and feedback, access to special events or exclusive swag, etc. as useful elements of a motivation program within a company. Obviously, this is all much more complicated than a simple technical solution that helps using a system in a secure way.

Account recovery

Grzegorz Milka illustrated the problem of offering technical solutions that are not adopted, using observations of account takeovers and recovery at Google. Accounts at Google are constantly under attack; the company goes to great lengths to protect account holders and to simplify recovery. His talk immediately made headlines in various tech outlets with the observation that 10% of Google accounts have two-factor authentication enabled. Some were stunned it was such a low number, while others were astonished that was so high. Whatever your perspective, what I thought more interesting was the general impression that Google has put a variety of methods into practice to prevent an account from being hijacked.

It's a risk-based, in-depth strategy that is complex and tries to keep up with the evolution of the account hijacking business. Devices are fingerprinted, the location of a login is taken into consideration and the known usage patterns of an account are part of the equation too. These observations do not stop once the account is recovered; the user behavior is constantly being monitored and abnormal behavior can trigger a lockout to prevent data loss or spreading to the contacts of a user via phishing scams.

The example he noted was a login from a new device, then a look at the address book, and then an attempt to erase all contacts. This resembles an account takeover where the attacker copies the address book for future attacks and immediately tries to make sure the account owner can no longer warn all his contacts. Google monitors the behavior of the user to try to detect malicious activity. This is necessarily far more complicated than simply providing two-factor authentication.

If you have a simple technical solution to a security problem, that is great. But if humans are not adopting it, then things get complex quickly. Simple technical solutions will no longer do the job and you need insight into social factors in order to understand human behavior and build your systems accordingly.

[I would like to thank the Swiss Cyber Storm conference for paying for my trip to USENIX Enigma.]

Comments (29 posted)

The rest of the 4.16 merge window

At the close of the 4.16 merge window, 11,746 non-merge changesets had been merged; that is 5,000 since last week's summary. This merge window is thus a busy one, though not out of line with its predecessors — 4.14 had 11,500 changesets during its merge window, while 4.15 had 12,599. Quite a bit of that work is of the boring internal variety; over 600 of those changesets were device-tree updates, for example. But there was still a fair amount of interesting work merged in the second half of the 4.16 merge window; read on for the highlights.

Architecture-specific

• The PowerPC architecture now has support for memory-protection keys.

Core kernel

• The expedited option has been added to the membarrier() system call. The API has changed somewhat since it was last discussed, though. At that time, there was a concern about not perturbing processes that did not need to participate in any given barrier operation, and some complex schemes for figuring out which processes to include. The merged patches, though, create an opt-in mechanism for processes that want to participate in "global" barrier operations. Opting in is a matter of calling membarrier() with the concise and easy-to-type MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED command. An expedited barrier can be sent to all registered processes with MEMBARRIER_CMD_GLOBAL_EXPEDITED.

Filesystems and block I/O

• Overlay filesystems using overlayfs can now be exported via NFS. Enabling this functionality changes some overlayfs behavior and can create a filesystem that is not backward compatible; see the documentation in this commit and this commit for details.
• The XFS reverse-mapping feature is no longer marked as being experimental.
• The AFS filesystem has "dynamic root" support; see this commit for more information.

Miscellaneous

• The new console_msg_format command-line option can be used to adjust the formatting of printk() output. In particular, setting it to syslog will cause a syslog-like format to be generated.
• Virtualization with KVM now supports AMD's secure encrypted virtualization (SEV) feature.

Security-related

• The usercopy whitelisting patch set has been merged. This mechanism further reduces the ranges of memory that can be copied to or from user space; now specific fields of a slab-allocated structure can be whitelisted. After some discussion, the current configuration will only issue a warning (rather than oopsing the system) if non-whitelisted memory is copied; the intent is to find and fix any problems without actually breaking the kernel when those problems occur. When the developers are confident that all of the issues have been found, that behavior may change in a more unforgiving direction.

  Anybody wanting stricter behavior now can disable the HARDENED_USERCOPY_FALLBACK configuration option or set the slab_common.usercopy_fallback command-line option to zero.

• Initial mitigations for Spectre variant 1 (bounds-check bypass) have been merged; see this article for details. The core of this support is a new array_index_nospec() macro that prevents speculation that might cause a bounds check to be circumvented.
• The arm64 architecture has gained another set of Meltdown/Spectre mitigations. The array_index_nospec() operator is supported natively, and it has been applied in a few places in the kernel. Branch-predictor hardening has been improved as well.
• S390 has also gained an implementation of array_index_nospec(), support for some new instructions to control branch prediction, and a variant on the retpoline concept called an "expoline".

Hardware support

• Clock: Hisilicon Hi3660 stub clocks, Spreadtrum SC9860 clocks, Qualcomm MSM8916 APCS clock controllers, AmLogic A113D clock controllers, and Freescale MXC realtime clocks.
• Pin control Meson AXG pin-control units, STMicroelectronics STM32MP157 and STM32F769 pin controllers, MediaTek MT7622 pin control units, Microsemi Ocelot pin control units, and Qualcomm MSM8998 pin controllers.
• Media: NXP TDA18250 silicon tuners, NVIDIA Tegra video decoder engines, Intel IPU3 CIO2 CSI-2 receivers, and OmniVision OV7740 sensors.
• Miscellaneous: Ilitek ILI9225 and ILI9322 display panels, OpenCAPI coherent accelerators, Cadence PCIe host controllers, Realtek RTD119x/RTD129x watchdog timers, and Spreadtrum watchdog timers.
• The old-style LIRC infrared controller drivers, having languished in staging for the requisite number of cycles, have now been removed from the kernel.

Internal kernel changes

• The way that printk() handles large amounts of message traffic has been changed. It will now spread the work over multiple CPUs, reducing the possibility of locking up a CPU with constant message traffic.
• The i2c-gpio fault injector can be used to create "unusual states" on the I2C bus to test the other side's error handling; see Documentation/i2c/gpio-fault-injection for details.

Barring a Meltdown-style security issue or some other problem of that magnitude, the final 4.16 release can be expected in early April. As always, though, there is a lot of testing and fixing to be done between now and then.

Comments (12 posted)

Shrinking the kernel with an axe

This is the third article of a series discussing various methods of reducing the size of the Linux kernel to make it suitable for small environments. The first article provided a short rationale for this topic, and covered link-time garbage collection. The second article covered link-time optimization (LTO) and compared its results to link-time garbage collection. In this article we'll explore ways to make LTO more effective at optimizing kernel code away, as well as more assertive strategies to achieve our goal.

Kernel module carving

Many kernel drivers start by allocating memory and registering stuff; then they sit there waiting until something they're responsible for happens, if ever. So the first course of action to reduce our kernel size is to make things into modules. This way some code can sit on the filesystem until it is actually needed.

Let's pick up the same kernel config and build environment as the one from the previous article, and reduce it by carving out some modules. We start with:

    $ make stm32_defconfig
    $ make vmlinux
    $ size vmlinux
       text    data     bss     dec     hex filename
    1704024  144732  117660 1966416  1e0150 vmlinux

Now let's enable module support and turn a couple things into modules:

    $ ./scripts/config --enable MODULES
    $ ./scripts/config --module INPUT
    $ ./scripts/config --module I2C
    $ ./scripts/config --module NLS
    $ ./scripts/config --module CRYPTO
    $ make olddefconfig
    $ make && size vmlinux
       text    data     bss     dec     hex filename
    1738586  137296  109180 1985062  1e4a26 vmlinux

This is bad. Despite the .data and .bss sections becoming 7,436 and 8,480 bytes smaller, respectively, the .text segment is now 34,562 bytes larger despite a couple drivers being removed from the core kernel. Is the module support code that big?

    $ size kernel/module.o
     text    data     bss     dec     hex filename
    12452     248      77   12777    31e9 kernel/module.o

So module support itself is only 12,452 bytes of text. Extra code for module support is expected, however there are still 22,110 additional bytes that appeared from somewhere. Could LTO help here?

    $ ./scripts/config --enable LTO_MENU
    $ make && size vmlinux
       text    data     bss     dec     hex filename
    1653358  137356  104648 1895362  1cebc2 vmlinux

LTO improved things, but produced only a 4.5% size reduction. We're far from the 22% reduction we obtained previously when modules were disabled. Why is module support so counter-productive?

The tree that hides the forest

We've seen that LTO is very good at optimizing code away. It is especially good at figuring out that some functions end up never being called; their removal means that even more functions end up not being called, and so on along the call graph down to the leaf functions. It is like cutting a limb from a tree; every sub-branch and leaf on it obviously won't be connected to the tree anymore and will fall to the ground. But what LTO does is more like getting rid of branches that simply float in the air without being connected to anything or which have become loose due to optimization. Branches connected to the trunk won't be trimmed. Neither will the trunk itself (the main() function) for obvious reasons.

The kernel, unlike user-space programs that typically have only one entry point, is different as it has multiple entry points. In fact it has so many entry points that we may compare it to a forest rather than a tree, with interlacing (interdependent) branches forming a dense canopy. No wonder why it is so hard to obtain a light kernel.

Among those kernel entry points we have:

• The start_kernel() function which is equivalent to the main() function in a user space program.

• Every system call (about 400 of them) is a kernel entry point. Obviously some of them must be present for the kernel to be useful.

• Every EXPORT_SYMBOL() statement is declaring an entry point to the kernel. Some of them designate data rather than code, but they create a dependency link just the same.

• Every initcall() instance, of which there are multiple variants, is yet another entry point of some sort, even though the caller remains within the kernel itself. Still, they add call dependencies of their own.

• And to a lesser extent in terms of code size, there are all those parameter parsing functions attached to early_param() statements.

Normally, the initcall() and early_param() instantiated code is marked with the __init qualifier and therefore evicted from memory once the boot process is complete. However, EXPORT_SYMBOL() really is a problem. Just by turning CONFIG_MODULES on, we added a lot of tree trunks to our kernel:

    $ wc -l Module.symvers
    3984 Module.symvers
    $ wc -l modules.order
    30 modules.order

In plain text, that means that there are 3,984 exported symbols in this kernel configuration for 30 configured modules. That means 3,984 additional entry points that LTO can no longer optimize away. It is unlikely that our modules need that many exported symbols. Let's find that out:

    $ find . -name \*.ko -exec nm {} \; |
    >      grep "^ *U" | sort | uniq | wc -l
    429

So 429 symbols are needed for our set of modules, including those symbols that are exported from other modules, meaning that in reality we'd need less than 429 exported symbols from the kernel core.

This is where the trimming of unused exported kernel symbols becomes especially useful. This is activated with CONFIG_TRIM_UNUSED_KSYMS, available since Linux v4.7. It works by gathering all required symbols from the set of configured modules (similarly to what is done above), and storing them in include/generated/autoksyms.h as a list of defines. Those defines control whether their corresponding EXPORT_SYMBOL() instance is activated or not. If adjust_autoksyms.sh detects that the list of symbols doesn't match the existing list, then it updates that list and triggers a rebuild of the affected source files. The process is repeated until the list of exported symbols becomes stable. Given autoksyms.h is initially empty, at least one rebuild loop is needed.

Let's see what we get from this:

    $ ./scripts/config --enable TRIM_UNUSED_KSYMS
    $ make
    $ wc -l Module.symvers
    429
    $ size vmlinux
       text    data     bss     dec     hex filename
    1546209  137292  109172 1792673  1b5aa1 vmlinux

Yes, we finally get a nice 9% size reduction. Let's activate LTO on top of that:

    $ ./scripts/config --enable LTO_MENU
    $ make
    $ wc -l Module.symvers
    294 Module.symvers
    $ size vmlinux
       text    data     bss     dec     hex filename
    1156851  136272  104512 1397635  155383 vmlinux

Not only did we get a 29% size reduction, but thanks to LTO, the compiler was able to optimize things so that in the end 135 fewer exported symbols were needed. Great progress at last!

Let's cut more trees

The next source of dependency trees, and potentially unused code, is system calls. Certainly we should be able to axe a couple of them. After all, our tiny user space to go along with our tiny kernel probably won't need most of them anyway.

The kernel configuration system already provides some options to enable or disable support for some system calls. Here are a few examples that our user space certainly can live without:

• CONFIG_SYSFS_SYSCALL for the obsolete sysfs() system call (not to be confused with the sysfs filesystem).

• CONFIG_ADVISE_SYSCALLS for the madvise() and fadvise64() system calls, which are not needed by many workloads.

• CONFIG_POSIX_TIMERS to remove most support for POSIX timers while preserving the simplest cases.

Turning off the above options produces this:

       text    data     bss     dec     hex filename
    1138567  135212  102404 1376183  14ffb7 vmlinux

That means a 1.5% size reduction compared to our previous kernel. This is not good if we need to rely on manual kernel configuration operations to get about 1% each time. What if we could scan our user space to automatically determine the list of required system calls just like we did for exported symbols? Such a scanning tool doesn't exist yet, but a simple lookup with objdump and grep on my busybox-based user space reveals 74 system call invocations, with some of them being duplicates of others.

Of course we care about the environment and don't want to cut healthy trees. But here we're talking about virtual trees that can be regrown with a make command. So let's just do some clear-cutting in the system-call forest to see how far this approach could go before investing more efforts in a proper solution. Let's apply the following hack to our kernel to get the compiler to simply remove every system call:

    --- a/include/linux/syscalls.h
    +++ b/include/linux/syscalls.h
    @@ -214,7 +214,9 @@
            asmlinkage long SyS##name(__MAP(x,__SC_LONG,__VA_ARGS__));      \
            asmlinkage long SyS##name(__MAP(x,__SC_LONG,__VA_ARGS__))       \
            {                                                               \
    -               long ret = SYSC##name(__MAP(x,__SC_CAST,__VA_ARGS__));  \
    +               long ret;                                               \
    +               if (1) return -ENOSYS;                                  \
    +               ret = SYSC##name(__MAP(x,__SC_CAST,__VA_ARGS__));       \
                    __MAP(x,__SC_TEST,__VA_ARGS__);                         \
                    __PROTECT(x, ret,__MAP(x,__SC_ARGS,__VA_ARGS__));       \
                    return ret;                                             \

Rebuilding with the above hack applied and LTO still active produces this:

       text    data     bss     dec     hex filename
    1050735  135208  102348 1288291  13a863 vmlinux

Removing all system calls reduces the kernel by a mere 7.8%. This is rather disappointing. It is probably not worth pursuing this angle for now.

Time to get dirty

We have probably gotten to the point where the return on the investment with automatic size reduction techniques is no longer worth the effort. Let's move to more involved approaches now, using explicit kernel configuration tweaking:

• Tiny embedded systems rarely have the luxury of a block device anyway, since they tend to be running directly on flash memory. So CONFIG_BLOCK=n.

• The STM32 system-on-chip is a no-MMU target; therefore any attempt to isolate users from each other is rather futile. So CONFIG_MULTIUSER=n.

• Some other goodies that we won't use anyway, and which can thus be disabled, include CONFIG_TIMERFD, CONFIG_MEMBARRIER, CONFIG_COMPAT_BRK, CONFIG_PROC_SYSCTL, etc.

• Let's select CONFIG_SLOB instead of CONFIG_SLUB; it should be sufficient for managing a small amount of memory, and it has smaller footprint too.

• With few user tasks, CONFIG_PREEMPT_NONE is most likely good enough. This has the nice side effect of enabling CONFIG_TINY_RCU and that sounds nice.

• We don't need all the kernel decompressors under the sun. So BZIP2, LZMA, LZO, LZ4, XZ are all gone.

And then we get:

       text    data     bss     dec     hex filename
     926250  129292  101172 1156714  11a66a vmlinux

Okay! A 17% reduction compared to our last working kernel. Still, it feels like we ought to be making much more progress at this point. We're 41% smaller than the kernel we started with, but this kernel is also less capable.

Time to get nasty

There is a tinyconfig make target that produces the smallest kernel possible. It doesn't boot on anything though. Still, this can be used as a starting point for yet more aggressive code modularization and axing. Let's have a look:

    $ make tinyconfig
    $ make vmlinux && size vmlinux
       text    data     bss     dec     hex filename
     508792   93844   20956  623592   983e8 vmlinux

This is getting nicely small now. However, this is still big for a kernel that, by definition, has everything configured out. Let's see where the bulk of the remaining code sits, with a little scripting:

    $ for f in */built-in.o; do
    >     size -t $f | tail -1 | sed "s|(TOTALS)|$f|"
    > done | sort -nr
     141183   15705    9377  166265   28979 kernel/built-in.o
     111627     816    5436  117879   1cc77 fs/built-in.o
      82893    5250    3493   91636   165f4 mm/built-in.o
      79197    3173    1788   84158   148be drivers/built-in.o
      65043     124      82   65249    fee1 lib/built-in.o
       5078   13894      61   19033    4a59 init/built-in.o
       2381       0       0    2381     94d security/built-in.o

That's our tinyconfig kernel. This is without LTO, otherwise it is much more difficult to get a size breakdown per subsystem like the above. But we've already seen that, even with a stripped-down kernel configuration, LTO has its limits. Regardless, this still raises some questions:

• Why is there still 77KB of driver code when there are no drivers configured in?

• Is it really necessary to entertain 80KB of memory-management support on a MMU-less target?

• Can we get rid of some of that 110KB of filesystem infrastructure when there is no need for a full-fledged filesystem support in our tiny system?

Digging into the lib directory we can spot some low-hanging fruit:

    $ size lib/crc32.o
       text    data     bss     dec     hex filename
      25528       0       0   25528    63b8 lib/crc32.o

The CRC32 checksum implementation is not small; fortunately, a smaller alternative is available:

    $ ./scripts/config --disable CRC32_SLICEBY8
    $ ./scripts/config --enable CRC32_BI
    $ make lib/ && size lib/crc32.o
       text    data     bss     dec     hex filename
        340       0       0     340     154 lib/crc32.o

Wonderful. Now could we do the same with the VFS layer with, for example, an alternative implementation of the dentry-cache code that has no hyper-parallelized algorithms that scale to thousands of concurrent users? Perhaps it could be just some dumb code that preserves the existing interface but which does the job fully serialized in the simplest way possible… providing another 75x smaller footprint reduction maybe? But there are other challenges to overcome before taking this approach.

When the winter hits the forest

Consider another possible opportunity for size reduction: the TTY layer. Digging into the drivers/ directory of our STM32 kernel, we can see this:

     text    data     bss     dec     hex filename
    59572    1447    2713   63732    f8f4 drivers/tty/built-in.o

Spending 60KB on kernel code and data to transfer bytes over a serial port — not counting the dynamically allocated memory — seems unnecessary in our tiny environment. An attempt at an alternative TTY layer to that effect was proposed (see the associated LWN coverage). Making more pieces of the kernel optional was also attempted. The scheduler is one part of the kernel where significant parts can be carved out pretty easily (also covered by LWN).

Unfortunately, those attempts were welcomed with a cold headwind that left me alone in the woods with frozen fingers.

A different angle

One thing that usually mitigates the wind-chill effect in the community is a working proof of concept. A prominent characteristic of tiny microcontrollers is an amount of on-chip flash memory that is typically a few times larger than the on-chip RAM. So to achieve our goal of running Linux solely from the on-chip resources, we should stuff as much code and data as possible into flash memory, and execute code directly from there while keeping actual RAM usage to the bare minimum. This has to happen for both the kernel and user space of course. Kernel eXecute-In-Place (XIP) has been available for quite a while, but it's another story when it comes to user space. That will be the subject of the next (and final) article.

Comments (23 posted)

Two FOSDEM talks on Samba 4

Much as some of us would love never to have to deal with Windows, it exists. It wants to authenticate its users and share resources like files and printers over the network. Although many enterprises use Microsoft tools to do this, there is a free alternative, in the form of Samba. While Samba 3 has been happily providing authentication along with file and print sharing to Windows clients for many years, the Microsoft world has been slowly moving toward Active Directory (AD). Meanwhile, Samba 4, which adds a free reimplementation of AD on Linux, has been increasingly ready for deployment. Three short talks at FOSDEM 2018 provided three different views of Samba 4, also known as Samba-AD, and left behind a pretty clear picture that Samba 4 is truly ready for use. I will cover the first two talks in this article, and the third in a later one.

War stories

The first talk, "SaMBa-AD, it works", was given by Denis Cardon from the French system integrator Tranquil IT. He specializes in deploying Samba 4, and was keen to tell of his field experiences in doing so. Samba has been big in France for some time, which Cardon attributes variously to the "free-as-in-beer syndrome" (because there are no per-seat client license costs with Samba), the "free-as-in-speech syndrome" (a general love of liberty), and "the General de Gaulle syndrome" (a dislike of American products being forced down French throats).

Samba 3 (which he referred to as "Samba-NT") has allowed some sites to stay clear of AD for much longer than a Microsoft NT-based domain controller infrastructure would have permitted; LDAP, SMB 2 and SMB 3 can all be used on a Samba 3 domain. But it's getting increasingly difficult to keep those sites running with newer client operating systems on the desktop; Windows 10 clients, for example, can only authenticate if you force the use of older protocols on the domain controller (DC). So over the past few years much of Cardon's business has been upgrading sites from Samba 3 to Samba 4. Of late, however, more and more sites are paying his company to come and migrate existing Microsoft AD deployments to Samba 4; word is spreading that Samba 4 is ready for production.

He has deployed Samba (both 3 and 4) in the French central government, in regional and city administrations, the French defense sector, schools, and universities. His clients include Samba 4 sites with five thousand desktops and ten thousand users, and sites with dozens of domain controllers; one site has 80. One client has asked about a 120,000 user deployment, but apparently Andrew Tridgell Bartlett, when asked, advised waiting for Samba 4.9, due out later in 2018.

Much of the rest of Cardon's talk was made up of amusing war stories: the site where the only working domain controller was a thirteen-year old NT 4 system the interior spaces of which, when opened, were entirely filled with dust. Deployments that had to be hastily finished before hurricanes took the power down, and those held up by coups d'état (though in one latter case, local representatives advised that the usual delay caused by a coup d'état was a week, so eight days later they reconvened and completed the migration). There was another migration where every system was successfully migrated except the computer that ran the door-entry-card system, so the following morning Cardon had to negotiate with security to break him into the site so he could fix it. Entertaining as these all were, the important lesson of the talk was that Samba 4 is ready to take on authentication duties for very large sites with complex requirements.

But Samba 4, as it ships with some modern server distributions, does not quite reflect this reality. CentOS 7, for example, released more than two years after Samba 4, does have a samba-dc package. But this package contains precisely one file, /usr/share/doc/samba-dc-4.6.2/README.dc, which explains that AD support is not available in Samba 4 as supplied by Red Hat. One can, of course, build Samba 4 from source, but this is not an approach favored by many production system administrators.

Vendor Samba

The second talk, "Samba AD in Fedora", given by the Red Hat Samba maintainer Andreas Schneider, explained the historical context behind the current state of vendor-provided Samba, and how things are changing. AD depends fundamentally on Kerberos, a network security protocol developed at MIT in the late 1980s. MIT Kerberos was for many years the reference implementation. Enterprise distributions, noted Schneider, are required to ship with MIT Kerberos if they want to do business with governments, and in particular with the US government. Everything in the distribution that is "Kerberized" will be integrated with MIT Kerberos. There is another implementation, Heimdal, but it is unwise to mix both implementations on a given system because the on-disk representation differs, and because there are symbol clashes if you try to load both libraries into a single application; the presence of one effectively excludes the other.

Samba, however, is built against Heimdal Kerberos. In response to a later question, Schneider said he did not ascribe this to US export restrictions on crypto. It was primarily because at the time the AD implementation was being developed, MIT Kerberos development was moribund. Heimdal, meanwhile, had an active community, which was open to implementing features that Samba needed. Volker Lendecke, who also spoke on Samba, noted from the audience that a similar situation had happened with LDAP: Samba needed an implementation, but the OpenLDAP community was resistant to accepting Samba-related patches, so Samba developed its own LDAP server. Thus, Samba went with Heimdal Kerberos. Meanwhile, Heimdal development had slowed dramatically (though it is now recovering), possibly because the lead developer went to Apple. MIT Kerberos development, however, has had a resurgence.

Those changes made for an auspicious time to try to make Samba work with MIT Kerberos. The first step, said Schneider, was to develop a test harness. Samba has an extensive test suite; Schneider noted some 15,000 tests in 2,000 test suites that take him about three-and-a-half hours to run. If code refactored to work with MIT Kerberos couldn't pass those tests, it wasn't going to get into upstream.

So, in February 2013, he started writing cwrap, "a set of tools to create a fully isolated network environment to test client/server components on a single host, complete with synthetic account information, hostname resolution, and privilege separation support". Or as he then put it more pithily, an equivalent of the Matrix for applications: a way to run applications while telling them lies about the environment in which they are running.

A year later, the team was ready to run Samba with cwrap, and started working on the refactoring. The developers also started finding, reporting, and, in some cases, fixing bugs, both in Samba and in MIT Kerberos. By April 2015, the patch set to Samba had grown to 140 patches. Rebasing had become a painful daily activity, not least because the team's branch of the code was generally ignored by other Samba developers, so the team had to keep up with everyone else's development as well. Most of the test suites were passing at this time, but 69 were still failing. Nevertheless, they started to push their patches upstream.

In April 2016, work was stalled for four months by the discovery of the Badlock bug. Just as that was fixed, MIT Kerberos 1.15 was released, and Schneider and colleagues discovered that the developers had removed the API function which allowed memory management for KDB modules, thus giving their refactoring a nice big memory leak. Hasty conversations with the developers about putting it back ensued.

By January 2017 they were down to a single test failing, and a couple of extant issues. These were fixed and, on 30 April 2017, the code hit the Samba Git master branch. Five months later, Samba 4.7.0, the first version with official MIT Kerberos support, was released. Work on the correct way to package this ensued, and on 14 November 2017 Fedora 27 was released—with Samba AD. No particular timeline was given for the move into RHEL, but that is how technology generally percolates through the Red Hat environment. Should you wish to run the Fedora 27 version, Schneider said, all the important stuff works (single domain controller, forest, and external trust relationships). Installation and provisioning is now just three lines:

    # dnf install samba-dc
    # samba-tool domain provision
    # systemctl start samba.service

The second command starts a dialog that asks for information specific to your site.

More advice on running Samba followed. Samba requires DNS, because AD does. It has its own internal DNS server, but it also provides a module to work with BIND, called samba-dc-bind-dlz. Schneider feels that the module is a horrible hack that directly manipulates the AD databases and thus has write access to them. His strong advice is not to use it, and to use the Samba internal DNS server instead.

AD administration in Samba is still fairly painful; Windows ADUC tools are probably the easiest way to do it. samba-tool is still fairly painful to use, and its error messages tend to consist of Python stack traces, but it is improving all the time. Lendecke was kind enough to demonstrate to me, there and then, the new "samba-tool user edit foo" functionality, which allows all of a user's information in AD to be seen and edited in one screen; new objects can be added and the tool complains properly if you try to add an object that is not in the schema. A limited but functional module for Cockpit has recently been written, which will at least allow you to provision and interrogate an AD domain. Patches are encouraged.

Some things are still not working. Audit logging is a work in progress, and should come in Samba 4.9. Smartcard support might still be missing; he thought it probably would work, but there are no tests for it, so he can't be certain ("we normally say at Samba ... untested code is broken code"). Read-only domain controllers are unsupported in MIT Kerberos and so in Samba, and quite a lot of work inside Kerberos would be needed to support that. That brought his talk to a conclusion, eliciting some well-deserved applause.

Schneider's talk can be seen in its entirety here and, for those who like system administration war stories, Cardon's talk can be seen here.

[We would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Brussels for FOSDEM.]

Comments (10 posted)

Authentication and authorization in Samba 4

Volker Lendecke is one of the first contributors to Samba, having submitted his first patches in 1994. In addition to developing other important file-sharing tools, he's heavily involved in development of the winbind service, which is implemented in winbindd. Although the core Active Directory (AD) domain controller (DC) code was written by his colleague Stefan Metzmacher, winbind is a crucial component of Samba's AD functionality. In his information-packed talk at FOSDEM 2018, Lendecke said he aimed to give a high-level overview of what AD and Samba authentication is, and in particular the communication pathways and trust relationships between the parts of Samba that authenticate a Samba user in an AD environment.

AD has been Microsoft's central user database since the days of Windows 2000. Before that, NT 4 had a flat user database. It had a GUI limit of 40,000 users and was clumsy to use when you got anywhere near that limit, but it worked. Microsoft moved to LDAP as the database and Kerberos as the authentication protocol, and AD was born.

The LDAP database is multi-master replicated, and suffers from all the problems that multi-master replication brings. The database itself has a highly-specific LDAP schema with custom extensions; native AD has a lot of internal magic and validity checks to ensure that the requirements of that schema are met at all times.

Authentication is challenge-response based, which is something he came back to in more detail later. AD also contains a DNS server element, and Microsoft used a standards-based approach to doing domain controller lookups using SRV records. AD allows trust relationships to be set up between domains. Lendecke noted that while it's typical for academic deployments to have a single domain across the entire university, in corporate environments it's quite common to find complex trust relationships between dozens or even hundreds of domains.

Samba started in the 1990s as a compatible file server for DEC PATHWORKS. It implements many protocols used by Microsoft: SMB for file sharing, SMB and DCE/RPC for print services, RPC for user database services, and Kerberos, LDAP, and DNS for the domain controller element. It is an NT 4-compatible domain controller, and both an AD domain controller and domain member, the latter being important in making AD user and group information available to Unix.

To avoid doubt going forward, Lendecke reminded attendees of the distinction between authentication (proving who you are) and authorization (having proven who you are, establishing what you are allowed to do). In AD, authorization is largely handled by group memberships: a user is usually a member of a large number of groups, each membership permitting access to a different set of resources.

Authentication

Lendecke then looked at different ways of authenticating by password, comparing the Unix transmit-the-password approach of FTP/telnet/SSH-by-password with Microsoft's challenge-response model. The critical distinction he sees is that, in the former cases, the password is transmitted from client to server, whether dangerously in plaintext (as FTP and telnet do) or safely under cover of an encrypted channel (as SSH does). This means that the server never needs to keep the password stored in plaintext. The password can be stored in an unrecoverable (hashed, salted) form, because the server will temporarily have the plaintext password offered by the client to salt and hash for comparison and verification.

In a challenge-response world, a nonce is transmitted from server to client. Both sides perform a similar operation on the nonce based on the password, then the client transmits its result to the server for validation. This requires the server to store the user password (or an equivalent) in plaintext for the long term. Lendecke notes that this allows the authentication server to impersonate any of its users to other machines, which is not true in a Unix world. He clearly prefers the Unix approach ("I always have to think very hard to make sure that Kerberos is actually safe").

In pre-AD Microsoft days, this authentication was accomplished by Microsoft's Challenge-Response Authentication Protocol. Lendecke noted this protocol completely deserved its acronym (CRAP) in the old OS/2 days, though by the days of NTLM v2 it had become quite good. Nevertheless, under this regime every single authentication event required communication with a domain controller. For certain workflows like HTTP authentication (with many short-lived requests), this could put a lot of load on the DC.

Under AD, authentication happens through Kerberos, which is a cryptographically secured ticket-based system. When you authenticate, the server gives you a signed ticket that you can then present to other servers when you want access to services. Those servers can verify your ticket without recourse to the DC. This means greatly reduced load on the DC, but the downside is that the protocol is extremely picky. Lendecke noted that all clocks must be in sync (though he accepted that is mostly solved these days), and that the DC must be contacted by its hostname. If you contact it by IP address, by a hostname mapped via CNAME, use a local /etc/hosts entry, or your DNS gets corrupted, Kerberos just doesn't work. Finally, under AD, NTLM must always be available as a fallback method in case Kerberos is not working. In response to an audience observation, Lendecke said he is often surprised by how many sites do not, in fact, have working Kerberos, but simply don't know it because everything is kept working by old-style NTLM authentication.

The presentation then turned to roles in authentication. Obviously, users have passwords and use them for authentication. But machines have passwords too, which they use to prove that they've been joined to the domain by an administrator. They can also use the DC's knowledge of these to validate the DC's identity: trust can be established in both directions. So, said Lendecke, enter Samba winbind, which is the daemon that is responsible for all of the traffic that he'd been discussing.

Winbind connects to the domain controller. In multi-site deployments it is important to be talking to a DC on the local area network, if one is available, rather than one on the far side of a high-latency link. The knowledge of which DCs are local is contained in the AD, but Lendecke said that it can be surprisingly complex to establish which is the best DC to talk to. Having selected an appropriate DC, winbind establishes an authenticated, encrypted channel to it for authorization purposes.

Lendecke noted that part of the protocol involves machines regularly updating their machine passwords, for security, and how complex that can be in a multi-master replicated environment. If the DC to which a machine has just given its new password dies without passing that on to the other DCs, or if the password change is accepted by all the DCs but the write of the new machine password to local disk fails, the machine will fall off the domain. AD implements a number of checks to prevent that from happening for Microsoft machines, so winbind has to as well.

Winbind provides an external interface through /tmp/.winbindd/pipe. This interface is extremely simple — some would say too simple — it has fixed-size requests (e.g. usernames can't exceed 256 bytes, but who cares, said Lendecke) and it just works. Samba provides PAM and NSS modules for hooking your Unix authentication to AD via winbind, for those who wish to do so. Lendecke noted that a design principle of winbind is that it tries to do exactly what Windows does and nothing else. That is all anyone can rely on, because anything else they try to do might break.

Authorization

Having discussed authenticating the user, Lendecke moved onto authorization. As he said earlier, access to files and other resources is primarily determined by the user's membership in groups. Because in AD (and unlike Unix) groups can themselves be members of other groups, this membership is a complex calculation. It is determined by the AD DC, and only when a user successfully authenticates. Any attempt to determine this important list of group memberships at any other time is prone to break.

This means, he said later, that while user foo authenticating and then doing id will see their list of groups, "sudo id foo" does not work and never can. It is simply not possible to calculate the list of group memberships reliably at any time other than when the user authenticates with their own password, not least because Windows doesn't calculate the list at any other time. As he'd said, Samba cannot reliably do anything that Windows doesn't do. People who don't get this are working under what Lendecke finds is a common misunderstanding.

So winbind handles authentication and authorization. But it also handles the mapping of Windows UIDs (which it calls Security IDs or SIDs, but these are fundamentally just rather oddly encoded 128-bit UIDs) to Unix 32-bit UIDs. There are an awful lot of different ways to do this mapping; Lendecke wishes that Unix supported 128-bit UIDs, because it would relieve winbind of having to do it. He noted he could give an entire talk about the remapping process — a talk that I would very much like to hear. Winbind also provides NSS information such as home directory and login shell to Unix.

Winbind's ability to do this depends on it being able to talk to the DC. It can talk to the controllers of the domain of which it is a member because it has a set of machine credentials that enable this. But given the inter-domain trust relationships that he mentioned earlier, it will need information from other trusted domains, and it has no credentials with which to get it. This means winbind must ask its DCs to get the information, which first means enumerating the trusted domains.

In the old NT 4 days this was simple, because only direct non-transitive trust relationships were allowed. But now trust relationships are transitive — if domain A trusts domain B, and domain B independently trusts domain C, then domain A effectively trusts domain C. Domain A's winbind daemons will need its DCs to access information from domains B and C. This is not simple, and has some hard-to-diagnose failure modes that can arise from things like misconfigured DNS and badly-configured firewalls sitting between different domains' DCs.

Finally, he reviewed winbind's NSS contribution. If Samba is going to handle authenticating users under Unix, it will have to provide Unix with the data that /etc/passwd and /etc/group have traditionally provided: a Unix UID, a primary GID and supplementary Unix group memberships, a home directory, a login shell, and so on. Most of these things do not normally exist inside AD, though with the Services For Unix (SFU) schema extension there is a place for them, and winbind will pass them back into Unix. Every user can now have an AD-stored Unix UID different from their Windows SID, and (as of Samba 4.6) an AD-stored Unix primary group different from their Windows primary group (which in most cases is simply, and somewhat unhelpfully, "Domain Users").

A video of the talk can be seen at the FOSDEM site. It's a dense talk, but for anyone who's trying to support a Windows network with free software, it's a highly useful one. At FOSDEM it is a frequent complaint at that the room is too small for the demand, but for the Samba talks (including the two covered in a previous article) demand was extremely high: by the end, there were many more people queuing outside the room for seats as there were seats inside the room. If it were possible to get a bigger room for these talks next year, that would be good, for I got the feeling the time has very much come for talks of this sort.

[We would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Brussels for FOSDEM.]

Comments (13 posted)