LWN.net Weekly Edition for January 19, 2012

LCA: A Samba 4 update

The systems administration miniconf at the 2012 linux.conf.au hosted 'a casual conversation' with a group of core Samba developers on the project's near future roadmap and the plans for Samba 4. Andrew "Tridge" Tridgell led off by saying that the last a lot of people had heard about the project's plans came from "an article in a disreputable web site." The discussion reported on there was "very exciting," in that it moved the project's point of view on the Samba 4 release from "someday" to "let's get ready for a release." Since then, things have gotten quiet, but that does not mean that nothing has been happening.

Andrew Bartlett took over to say that both he and Tridge think that the project is about ready for the Samba 4 release. The active directory (AD) domain controller (DC) support - a headline Samba 4 feature - is working well and is in production use in a number of sites; it is time to get it out there to the rest of the world. While they think that, at this point, things are ready for a release, the idea came as a shock to some of the other members of the team. Samba 4 had been seen by those developers as being far out on the horizon; they were not expecting talk of a release at this point.

The ensuing discussion was lively, but AD DC support was not the main point; everybody seems to agree that it is working well. The sticking point has to do with the long-time "bread and butter" features of Samba - little things like file serving. The new file server implementation in Samba 4 is missing a number of features that have gone into Samba 3 in recent years, so now the focus is on integration of Samba 4 with the Samba 3 file server. The developers have come up with a plan for this integration, and are now busily trying to implement it as quickly as possible. As Tridge put it, they ran into a social problem and came up with a technical solution because, in the end, coding is easier than arguing. The discussion has gone quiet because this coding is underway; they expect to present their solution soon, at which point the release discussion can be expected to restart.

Andrew spent some time talking about some of the things the Samba team has achieved with Samba 4. One of those is the new integrated build system - there is now "a single Samba." It is possible to build all binaries together; and there are a number of plugins to further integrate Samba's various pieces. As a result, Samba is now "one project," rather than a collection of related pieces.

Related to that is the new combined testing framework which is, according to Andrew, the most important thing that the Samba team has achieved. The framework can do full testing of all AD semantics. It is also set up to test Samba 3 and 4 against each other. A number of "rather embarrassing" interoperability problems between the two releases have been found and, naturally, fixed. This testing can now be done before every commit.

There is also a common security system that simplifies administration and fixes a lot of old "misunderstandings of Kerberos" that have been with the project for a long time. The Samba 3 and 4 security architectures have been merged.

All of this, Andrew said, has been good to make the new system work well, but it does not necessarily change the user's experience of Samba. There has been new feature work done as well, though. At the top of the list, according to Tridge, is subdomain support. Lots of sites do not work with a single domain at this point; instead, they have "forest" of domains organized into a hierarchy. Getting Samba to work in this mode has taken a lot of work over the last year. The 2011 plugfest event, where eight or so Samba developers went to Redmond to work on interoperability issues with Microsoft, was dedicated to firming up subdomain support and getting to a point where Samba can work at any level in an AD forest. It does work, but has not yet been designated ready for production; Tridge said he would like to see a couple of "brave" production sites deploy it and let them know how it works for them.

The project's relationship with Microsoft, they said, is quite good. They get quick answers to questions, even for detailed protocol history queries that require a fair amount of digging in the code to answer. Tridge said that he has been very impressed with the quality of the engineers that Microsoft has assigned to work with the project.

Another area of development is easing the process of upgrading from Samba 3 to Samba 4. Production sites, it seems, do not react well if you tell them that all of their users have to set new passwords before they can work under a new Samba release. At this point they have full user and group import into Samba 4, so users should not see the difference. The update is transparent to clients, except that they see the new AD support and start using it. There is still a bit of a flag day involved, though, in that clients, once they see an AD server, will not go back to talking to an older server release. So careful testing before deploying Samba 4 is still called for.

Amitay Isaacs and Kai Blin talked about their area of work: the built-in DNS server. Amitay has implemented one solution, whereby a new DLZ plugin for bind9 enables it to get its domain information from the AD database. It works, but it is "a bit clunky" as a result of the interactions between the two separate subsystems. So Kai is working on a new, internal DNS server. He had tried, he said, to get an existing DNS server project interested in closer integration, but found no takers. So he wrote a new server which, he said, was not that hard a problem. It is working now, with signed updates being the main missing feature at this point.

The "roadmap," according to Andrew, is that Samba 4 will probably be the next release from the project. It will include all of the expected features, including file and print servers, support for NT4-like domain controllers, and active directory support. It will also feature a number of improved tools and better usability in general. Samba has seen nearly 8,000 commits over the past year, changing 800,000 lines of code, and coming from some 70 authors. It has been, he said, a busy and important year. With a Samba 4 release likely, 2012 could be an even busier and more important year for this project, which quietly celebrated its 20th anniversary at the end of 2011.

[Your editor would like to thank the LCA organizers for assisting with his travel to Ballarat.]

Comments (13 posted)

Tizen releases source code and SDK previews

The nascent Tizen project unveiled its first set of materials on January 9, consisting of "preview" releases of the operating system source code and SDK, both intended to elicit feedback from developers. The announcement was accompanied by the launch of two new mailing lists and online documentation of the project's architecture and APIs. Tizen was announced in September 2011, but Intel took a markedly different approach to publicizing and promoting the project than it had with MeeGo. While MeeGo was launched with fanfare, the Tizen governance structure, policies, community resources, and ultimately code have been delivered at a much more measured — if not downright conservative — pace. Critics (particularly those upset over the demise of the MeeGo project) labeled the new project "vaporware" early on, but they — and everyone else — can now examine the source code up close.

Sources

The source code release comes in the form of a git repository, which hosts both Tizen packages and trees pulled from upstream projects. Tizen is often referred to as the successor to MeeGo, which was an Intel-Nokia collaboration on a "consumer electronics" Linux base layer derived from merging the two companies' existing projects. But even though Tizen's goals are the same, the actual software is a significant departure from MeeGo — the stack is based on the LiMo Foundation's work and is expected to integrate Samsung's Bada platform. More significantly for third-party developers, the application framework is centered around HTML5 rather than Nokia's Qt.

The components beneath that application framework are quite similar, of course (as they are in most embedded Linux distributions): the kernel (version 2.6.36), EGLIBC (2.13), Busybox, D-Bus, PulseAudio, GStreamer (0.10), ConnMan, and so forth. Among the other notable modules are a multi-lingual input framework derived from SCIM, hardware acceleration for OpenGL ES 1.1 and 2.0, and a WebKit-based web runtime. There is also a security module which provides Smack access control, secure data storage, and certificate management.

Perhaps the most significant departure from MeeGo is that Tizen's graphics toolkit is provided by the Enlightenment Foundation Libraries (EFL), rather than GTK+ or Qt. It has been a few years since Enlightenment and EFL were headline-makers on Linux desktops, but they have been in active development all the same, with support from Samsung, among others. LWN looked at EFL back in August.

EFL provides a canvas (called "Evas") that supports multiple back-ends; the Tizen project's is currently using the X11 back-end, which came as a mild disappointment to some observers, considering the work Intel had put into porting MeeGo to the Wayland display server. EFL's Carsten Haitzler told the mailing list that the hurdle was a lack of Wayland drivers for the system-on-chip video hardware targeted "at this stage," however, so there remains hope for Wayland support in the future.

EFL also provides application developers with a scripting environment called Embryo, a widget set (with touch support) called Elementary, as well as various utility libraries for timers, callbacks, event loop management, and data structures. Tizen includes the E17 version of EFL's Enlightenment window manager which supports compositing.

Web APIs

EFL does not encompass as much as Qt, however — richer APIs for hooking into system services and support for device features (like cameras and sensors) are both provided through Tizen's HTML5 application framework. The framework consists of three pieces: the "Tizen Web API," the Wholesale Application Community (WAC) "Device APIs," and the World Wide Web Consortium (W3C) "Widget Specification."

The Tizen Web API covers generic application objects, and interfaces to the communications, system information, and personal information management (PIM) subsystems. The supported communication frameworks include Bluetooth, NFC, cellular phone calls, and a general "messaging" module that encompasses email, IM, SMS, and MMS. The Tizen Web API's Contact, Calendar, and Messaging modules are each extensions of the corresponding WAC Device API modules, apparently with minor additions only. For example, The Tizen Web API's MessagingMessage adds message IDs and inResponseTo attributes, which allows an application to thread conversations.

WAC is an organization formed in 2010 by telecommunications companies and device OEMs. It publishes its own suite of OS-independent application development APIs, which are intended to provide a uniform HTML5 target for developers across the members' products. This effort is similar to the work produced by the W3C's Device APIs Working Group, but WAC does not develop its APIs in the open. Further confusing the situation, some of WAC's APIs require compliance with the corresponding W3C API, but some do not.

The Tizen alpha makes use of WAC's PIM APIs (the contacts, calendaring, and messaging modules mentioned above), its sensor APIs, its camera module, and its filesystem module. The sensor APIs provide interfaces to a mobile device's hardware sensors: accelerometer, digital compass, ambient light sensor, rotation/orientation sensor, and so on. The camera API covers both still and video capture, as well as displaying a preview image on screen. The filesystem module provides a single view of a device's storage, combining all on-board or removable storage into a unified virtual filesystem.

Finally, in spite of its name, the W3C Widget Specification is actually a standard for "installable" web applications — a storage format for compressed HTML5, CSS, and JavaScript applications, with an XML-based manifest. The Widget specification supports digital signatures and a device-controlled security policy to limit applications' access to network resources and local data.

Together, the Web API offerings may sound like a bit of a hodge-podge — after all, the WAC and W3C specifications overlap in many places, and conflict in others. Intel's Sakari Poussa responded to one developer's question by saying that Tizen was tracking the standards as best it could, considering that both APIs are still evolving, and that the project was committed to sticking with existing standards where possible.

Developing

In addition to the source code, the alpha release also includes the first look at the Tizen SDK. The SDK is available only for 32-bit Ubuntu (11.04 and newer), though the project says other distributions and operating systems will be supported in the final release. The SDK download is an installer script, which fetches and installs an Eclipse-based IDE, a QEMU-based emulator, testing tools, and support files.

The IDE includes project templates in two classes: "Tizen Web projects" and "native" Tizen projects. That is more-or-less in line with the messaging of the project, which emphasizes that HTML5 is the only supported application framework, but expresses tolerance for developers who wish to write code directly for the system APIs. The web project templates come in several flavors: WAC-only, generic HTML5, and jQuery (in several variations), and include Tizen packaging rules. The IDE experience also incorporates an event injector supporting mobile device hardware events, sensor data, and voice or SMS traffic.

Some web application debugging tools are also built in to the IDE, such as a WYSIWYG HTML5 editor and JavaScript/CSS syntax checkers. I was never able to get the IDE to launch the web project templates without crashing, so I cannot speak to their usefulness. There are known issues with the SDK mentioned in the release notes (including the fact that it is incompatible with OpenJDK — my experience suggests it may also be very picky about Sun Java 6); hopefully these concerns will be addressed between now and the final release.

The emulator has a few known issues of its own, of course, but the basic functionality is there. Currently only x86 targets can be created, with a handful of configuration options. The form factor of the virtual device is a keyboard-less mobile phone. The SDK documentation emphasizes that smartphones and tablets are the target devices for the alpha release, but that others will follow. Like MeeGo, Tizen is designed to eventually serve several distinct classes of hardware (including vehicles and smart-TVs).

The SDK also includes a documentation browser that encompasses an overview of the platform, the SDK tools themselves, and the various APIs. Some of this information is replicated on the Tizen developer site: the "Getting Started with Tizen" and "Tizen Web API Reference" sections in the documentation. But there is more in the SDK's browser, which covers packaging issues and hacking on the Tizen platform itself.

Community communications

Although it was not part of the developer preview per se, the SDK and code release coincided with the launch of two new Tizen mailing lists: one for application developers, and one for Tizen contributors. Up until now, the Tizen-General list and IRC channel had been the only communication channels for the project — very different from the approach taken with MeeGo, which had numerous mailing lists, forums, IRC channels, and a blog-aggregating Planet. Intel's we'll-announce-nothing-until-it-is-ready approach with Tizen has produced a slower trickle of discussion, but perhaps that is better than the publicity blitz that accompanied MeeGo's launch, when you consider that announcing the project and putting devices in consumers' hands can be many fiscal quarters apart.

A related development on the project management front was the sudden disappearance of the LiMo Foundation web site, which was replaced by the Tizen Association on or about January 1. The Tizen Association is essentially a re-branding of the LiMo Foundation, and, as yet, Intel itself has not finalized its membership. The Association's site describes its goal as enabling "key stakeholders to actively shape the industry role of Tizen and develop its market presence" by the "gathering of requirements, identification and facilitation of service models, and overall industry marketing and education". The project itself will continue to be hosted by the Linux Foundation.

The specifics of Tizen's project governance have not been fleshed out, but those are probably details that should come after the code itself has been released and developers have had a chance to work with it. In retrospect, the MeeGo project was very organization-heavy (as it was marketing-heavy), and in the end that did not help it make an impact in the marketplace. Tizen may still be a long way from shipping on commercial devices, but starting with the code rather than the other trappings of a large distributed project is a good first step.

Comments (3 posted)

LCA: Addressing the failure of open source

Bruce Perens wore a suit and tie for his linux.conf.au 2012 keynote for a reason, he said: it reflects our community's need to think more about how it appears to the rest of the world. Despite our many successes, he said, we have failed to achieve the goals that our community set for itself many years ago. We have failed to engage and educate our users, and are finding ourselves pulled into an increasingly constrained world. To get out of this mess, we will have to make some changes - and expand our scope beyond software and culture.

The open source (he always used that term) movement's goal, he said, was once to help a population that increasingly does everything - from entertainment to finances and voting - through computers. We wanted to enable people to do, not to be done to. But we find ourselves in a world where people are increasingly slaves to their tools. Yes, he said, tools like the iPhone empower their users, but they also constrain those users. They are designed not to allow their users to do things that might reduce the profits of their makers or of a number of related industries.

Why is this important? It matters to anybody who likes democracy. Almost all information is mediated through computers now - through computers controlled by a very small number of large corporations. That is a threat to democracy - and to freedom. But, he asked, why is it open source's job to address this problem? The answer is that the open source community is the only credible producer of software that is not bound to a single company's interest. History has given us absolutely no reason to believe that we can trust companies or governments to do that for us.

To be able to fill that role, we must continue to establish and defend our right to write and distribute open source software. In other words, he said, open source software should be legal. Things could be worse: we have been very lucky with software patents, for example. Patents could have been used to shut us down entirely by now; that has not happened - yet. But, increasingly, we find ourselves shut out of access to content, which does not help our cause.

There are also threats from new laws; increasingly, Bruce said, general-purpose computing is seen as a threat. Anti-hacking laws threaten the security of the net as a whole. SOPA-like laws threaten our tools and our infrastructure. The advent of 3D printers is going to inspire a whole new raft of laws as politicians figure out that they can be used to create weapons or other things that they do not want us to have.

We are very much on the defensive in these battles; why, he asked, do we have so much trouble being heard? The answer is that open source software has not built a relationship with "the common person" and does not have their sympathy. Instead, open source software has been co-opted and is being used to control people; thus we end up with locked-down products like the TiVo. We are, Bruce said, helping to make people into slaves of their tools. A project he worked on - busybox - is now shipped as an important component in any number of locked-down systems.

As an aside, Bruce said that, in retrospect, he should have continued to lead the busybox project.

It comes down to the fact that open source has not been able to protect its future through the reform of hostile laws. That is a result of our failure to reach the common man. We, as a community, have not achieved a sympathy for our users in the way that companies like Apple have. Some specific projects have had some success - he named Mozilla and Wikipedia - but, in general, our work is not particularly accessible to - or visible to - most users.

In fact, he said, most of us do not really even like users; we are making the software for ourselves. Demands from users who have not contributed anything are just annoying. But, he said, we need to get over that mindset; working only for ourselves is too limiting.

What about Ubuntu? It is, according to Bruce, demonstrating the necessary discipline to reach users. But, unfortunately, Ubuntu has turned itself into the intermediary between open source and its users; other open source companies, such as Red Hat, have done the same thing. We need, he said, to be more honest with ourselves about our partners. Commercial partners will always put their own business interests above any other consideration - in many countries, public companies are required to do that. They cannot work for the best interests of open source software.

So, he asked, where did we go wrong? There was a period where we held a moral high ground; that enabled political progress that we are not making any more. As for-profit corporations took over the foreground, we lost that high ground. Now we are just competitors, like any other. The commercial intermediaries represent us now, but they represent us in their own best interest. That is not surprising - it is what is expected of them - but it shows why they are wrong for this role. We really only have one tool for managing the behavior of these companies - copyright - and it is not helpful in this situation.

In dealing with companies, Bruce said, we always need to keep track of whether we are getting back something worthwhile for our efforts. Working for free to enrich Mark Shuttleworth is not a smart thing to do. We need to maintain a presence that is independent of organizations like Ubuntu. Debian, he said, does the real work of making Ubuntu; Canonical just adds the frosting. But does anybody in the wider world care about Debian at this point? We are, he said, failing at self promotion. Yes, tooting our own horn is tiresome, but if we don't do it, nobody else will do it for us.

Mobile apps are an especially big problem, he said. This is a commercial paradigm that has succeeded; that means, in particular, that the web failed. Now we are seeing open source software being sold as apps to people who don't know that they are buying something they could have for free. These apps, he said, duplicate every problem we have seen with proprietary software, only worse because these apps are meant to be ephemeral. We are going to have to find a way to change this paradigm, somehow.

Concluding this part of the talk, Bruce noted a strong trend away from generic platforms, toward proprietary, locked-down platforms. In the end, we will be left with nowhere to run free systems, and only jail environments for applications. Open source software, he said, has achieved tremendous things. But he is seeing signs that we have peaked; the locked-down platform is beating us. If we cannot find a way to address this problem, he said, it is our destiny to live in a constrained world.

Open hardware

There is a moment, far into Arlo Guthrie's epic "Alice's Restaurant," where he is required to remind the audience that, contrary to what they may think, they are listening to a song about Alice. Bruce's talk was similar; it was billed as being about open hardware, but it was only toward the end that he actually got around to talking about that subject. It is clearly Bruce's hope that a new revolution is building around open hardware, and that it may, in the end, help to save us from locked-down platforms. There are a lot of challenges in this area, he said, but he reminded the audience that the open source community has done many things that people thought were impossible.

Once upon a time, he said, people working on open hardware might have been seen as "isolated nutjobs." The good news is that, in the age of the Internet, there are no more isolated nutjobs. No matter how nutty you may be, there will always be fifty others like you on the net. In the open source world, those people got together and created tools that others could build on; things then took off from there. This history is, he said, repeating itself with open hardware.

There is a new set of enabling factors that are driving open hardware. We have the net, of course, but we also have the economic paradigms and collaboration methods developed by the open source community. But the big thing is that hardware is slowly being transformed into software; it can be developed - and shared - in much the same way. 3D printers, increasingly, let us print out solid objects that can be designed like software. Gate arrays allow the immediate implementation of complicated digital logic - at the level of a new CPU, for example. Throw in fast, cheap circuit board manufacturing and there is a lot that can be done by just about anybody.

So what is happening with open hardware? For many, the flagship project is certainly Arduino, which has found its way into many or most open hardware projects at this point. The boards can be cheap enough to (for example) throw into the LCA swag bag. There are over 250 "shields" adding interesting functionality; for any problem, there is likely to be a plugin solution out there somewhere. It has a development environment that makes it programmable by people like artists; as a result, Arduino is showing up in a lot of art applications now. And the whole thing is surrounded by an active community.

Arduino has some down sides. It does not use a 32-bit processor, so it cannot run complex software (like Linux, for example). There are a lot of alternatives out there, though, for those needing more power; the Beagleboard and Pandaboard offerings for example. Bruce also called out the Omap L138 processor in particular, calling it a dream for mobile software-defined radio applications. It has four CPUs, a lot of peripherals, and is quite affordable.

Bruce waved around a DSO Quad, a pocket-sized digital storage oscilloscope. It is all open hardware and software; it shows that one does not need a large corporation to create compelling mobile products. Then, there is the Bus Pirate, a general-purpose electronic hacking tool that can take on all kinds of tasks from programming FPGAs to sniffing data transfers from an I2C bus. Bruce also mentioned the Papilio gate array development board. This device can become almost any kind of circuit from a CPU emulator to a receiver for 20 simultaneous radio streams.

For those wanting more physical projects, there is the MakerBot, a plastic 3D printer. MakerBot leaves a lot to be desired; in particular, Bruce said, it will fall short until it has enough resolution to make high-quality Lego parts. It will be able to make anything one might want eventually; for now it is limited to "small, crappy-looking things." But it shows where things are headed. For the truly adventurous, there is OpenPCR, which allows genetic sequencing and duplication at home.

The list could probably have gone on for a long time, but Bruce had made his point - there is a lot happening in the open hardware area. There is the potential for amazing things to happen. But much depends on the legal climate, and it is clear that there are going to be legislative attacks on open hardware. We will have to be attentive if this revolution is not to be choked off before it can really begin.

[Your editor would like to thank the LCA organizers for assisting with his travel to Ballarat.]

Comments (157 posted)

Help wanted at LWN

Do you have an interest in helping to create the best free software technical information site on the net? Here at LWN.net, we have reached a point where we can hire another full-time editor if we can find the right person. We have put together a list of the skills we are looking and if you (or someone you know) meets them, please get in touch. The full job listing has more information.

Comments (none posted)