Leading items

Extracting the abstract syntax tree from GCC

Richard Stallman recently revived a nearly year-old thread in the emacs-devel mailing list, but the underlying issue has been around a lot longer than that. It took many years before the GNU Compiler Collection (GCC) changed its runtime library exemption in a way that allowed for GCC plugins, largely because of fears that companies might distribute proprietary, closed-source plugins. But efforts to use the plugin API to add features to another GNU project mainstay, Emacs, seem to be running aground on that same fear—though there has never been any real evidence that there is much interest in circumventing the runtime library exception to provide proprietary backends to GCC.

Last March, a debate about exporting the abstract syntax tree (AST) information from GCC to Emacs (and other programs) was ongoing in various mailing lists. That discussion was about adding an Emacs feature to auto-complete various program-specific identifiers (e.g. variable names, types, structure member names, etc.). It was an offshoot of another wide-ranging discussion that we covered back in January 2014. When the conversation dropped last March, Stefan Monnier had responded to Stallman:

On January 2, Stallman renewed the conversation by noting that he hoped "we can work out a kind of 'detailed output' that is enough for what Emacs wants, but not enough for misuse of GCC front ends". He was looking for people to help define that "detailed output", but instead found a number of people that felt that exporting the full AST information would be more sensible.

Stallman is concerned that proprietary backends could take the AST output and generate code from it. While no one in the thread wanted to see that happen, most also saw it as an unlikely outcome. David Engster said that he had been working on a way to get the AST information out of GCC for Emacs and noted that there was no technical barrier to doing so:

While the original discussion was largely about auto-completion, Engster and others would eventually like to go further than that. Their vision is to turn Emacs into a more full-featured integrated development environment (IDE), which would require all of the AST information, at least in their eyes. Stallman would prefer providing far less information: "just enough to do the completion and other operations that Emacs needs, and NO MORE."

Engster replied that he understood Stallman's concerns, but felt that there was no real problem:

He went on to say that if Stallman was opposed to using the AST in Emacs, he would drop the work he was doing in that area (for auto-completion and other IDE-like features). Stallman remained unconvinced of the need for the AST information, saying "it is so important to avoid the full AST that I'm not going to give up on it just because someone claims that is necessary".

But Perry E. Metzger noted that he is doing "a bunch of complicated refactoring work in conjunction with my current academic research" and that the lack of AST information from GCC forced him to use Clang/LLVM. He would like to see Emacs gain more IDE-like features that other tools, such as Apple's Xcode, have:

Stallman would like to see some kind of investigation to determine what pieces of information are needed for which purposes (e.g. auto-completion, refactoring, and so on), but that seems short-sighted to some. Metzger gave several examples of things that could only be done easily by having the full AST. Object-oriented languages, in particular, have programs with complex inter-relationships that can only be untangled by using all of the information that the compiler has collected in the AST.

Even if various subsets of the AST information could be defined to enable particular IDE features, new IDE features come about regularly. As David Kastrup pointed out, adding new interfaces for each new set of requirements makes little sense and would require that the Emacs IDE features and GCC versions be tightly coupled. Either that or the GCC plugin interface needs to be stable and provide a highly general API into the guts of the compiler, which could also be used in "bad" ways:

There appear to be a few disconnects in the conversation. Stallman is focused on auto-completion, and believes that it can be done without access to the full AST. Others disagree, especially for languages with operator overloading such as C++. Stallman said that he has never used C++, so he is trying to understand what would be needed to support auto-completion for C++ in Emacs. However, the discussion has also included adding support for more than just auto-completion, but Stallman is not ready to look into additional features yet. Even if it is possible to handle C++ (and the other languages GCC supports) auto-completion without all of the AST information, there are plenty of examples in the thread (mostly from Metzger) of IDE features that do require the AST (or enough information in other forms that it would be functionally equivalent to the AST).

It is clear that Stallman has not used any of the "competing" IDEs (e.g. Xcode, IntelliJ IDEA), which is not a huge surprise, but he is clearly feeling browbeaten about the issue: "Rather you are trying to pressure me to do what you want, at the expense of something I consider important." But Metzger and others have clearly stated that they understand (and, in general, agree with) Stallman's concerns, they just see the tradeoff differently than he does. In fact, Metzger said, there is a freedom issue at stake:

That is, of course, a hot-button issue for Stallman, who takes umbrage at that characterization. Furthermore, he wants to take some time to study the problem(s), without being pressured:

Metzger volunteered to help with the process, but noted that it goes well beyond auto-completion. Emacs is a powerful tool that, unlike most of the other IDEs, gives its users the ability to reprogram the way it works. But that requires flexibility:

Beyond that, Kastrup and others felt that Stallman was being unfair to Metzger by characterizing his comments as "changing the subject". Karl Fogel pointed out that Metzger and others have all acknowledged Stallman's concerns, but that Stallman has not done the same:

It would seem that the goal of more IDE-like features for Emacs has suffered a setback. Based on Stallman's responses, Engster said he would not continue working on his project to incorporate AST information into Emacs:

The problem that Stallman foresees is that the AST information could be used by a non-GCC backend that wouldn't use libgcc and would thus evade the GCC plugin restrictions that were added to the runtime library exemption. But as Óscar Fuentes pointed out, that has already been done by the DragonEgg project, which "is now abandoned, mostly because Clang is a better front-end than GCC". Because LLVM does not have the freedom-respecting requirements that Stallman holds dear, it also takes a much more modular approach that is easier for other projects to interface with.

Another potential problem is the possibility of a fork of Emacs to support using a GCC plugin that exports the AST, which would be legal under both the runtime library exemption and the Emacs license. The only barrier would be if Stallman was unwilling to accept the code into Emacs. Monnier said that, under those circumstances, he would be "willing to consider a fork if that's what it takes".

Kastrup noted that Stallman has laid out a plan for getting the understanding he needs to proceed with these features. That will take time, however, which may have other negative impacts:

There appears to be a fairly wide chasm between the two sides of the debate. It is hard to see how an Emacs IDE mode for GCC can compete with the proprietary alternatives without some mechanism to extract the AST from the compiler. To Stallman, at least, that is not of paramount importance, while others see things differently. The risk is that both Emacs and GCC decline in both usage and developer mindshare while some kind of solution is found. That would seem to make it worth coming to a solution sooner rather than later, no matter which side of the debate one is on.

Comments (72 posted)

Bob Young on freedom, control, and the GPL

Bob Young, known to the free-software community as the co-founder of Red Hat, founder of the print-on-demand service Lulu, and creator of the non-profit Center For The Public Domain, delivered the morning keynote address on the first full day of linux.conf.au (LCA) 2015 in Auckland. Although Young confessed several times to not being as plugged-in to the Linux and free-software economy as he once was, he had plenty of wisdom to dispense to the crowd, in particular by looking back on which (perhaps unexpected) factors allowed Red Hat to become a successful enterprise in the face of large competitors selling proprietary software.

Young's talk was laid-back, informal, and peppered with wisecracks and comedic stories—he introduced himself as "the light entertainment interlude" between the week's other keynotes from Eben Moglen and Linus Torvalds, and he consistently referred to himself as "just a typewriter salesman," to pick out two examples from many. But whatever description one might use, Young oversaw the rise of Red Hat in the early 1990s from a small software shop to a major IT vendor, and at the start of that process, it was far from a proven strategy that free software could compete for customers in the commercial space.

Young said he was initially a bigger skeptic about free software than Bill Gates, but while he was editing an early software-development newsletter, he repeatedly heard from developers that free software was what they wanted to hear more about. By the time Linux 1.0 was released, he decided to find out for himself how it all worked, so he went to meet with Richard Stallman, Donald Becker, and other "really smart guys" and figure out why altruism—an idea that Young had never thought would be capable of spawning a sophisticated technological system—seemed to be succeeding.

Each of those encounters had its own lesson: Stallman advocated the egalitarian notion that each free-software developer contributed according to their ability and reaped rewards in accordance to their skill level; Becker argued that sharing software was simply "the right thing to do;" Becker's boss Thomas Sterling explained the notion of the barter economy, saying "I contribute a driver and get back a gigabyte of software with the license to do what I want with it—and you think they are exploiting me?" But what finally clicked for Young seemed to be what he heard from engineers: that free software gives them control over their own system, and control was the one thing that proprietary software vendors were unwilling to part with.

That idea was what led to the formation of Red Hat. It was fundamentally against the proprietary IT vendors' business models to sell a product that allowed the customer to have control, so Red Hat had an immediate competitive advantage against all of them. Young added that he recently stopped by Red Hat's offices (his first visit in a long time) to try and ensure that he didn't put something problematic into his LCA talk that would cause the company's stock price to plummet. On that visit, he said he was thrilled to see that the corporate messaging remains essentially the same: customers are treated as partners with the company; giving them control over their systems is still paramount.

Young then spoke briefly about software licenses. He has long been a staunch advocate of the GPL, he said, but not because he has detailed opinions of the differences between it and other, similar licenses. Rather, he said, he learned early on that customers do not care about the details of free-software licenses, and that diving into an explanation of them was the fastest way to send a potential customer running toward the competition. (In an aside, Young made it clear that even during his days at Red Hat, he never considered other Linux distributors to be "the competition;" he used the word only to refer to the proprietary vendors.)

With the GPL, everyone knows what the rules are, more or less, so the debate is over quickly. They might not actually know the rules, he added, but they think they do, which amounts to the same thing. As a "business guy," he said, he is always trying to simplify the "pitch"—the GPL tells everyone "what we're about," so it has the greatest impact. Furthermore, he advised developers not to waste any time trying to come up with a new, specially-crafted free-software license for their project. Whatever effort it takes to write that new license corresponds to creating more complications for the potential customer. In the end, it is a net loss.

Young took a few questions from the audience at the end, some about Red Hat and some about Lulu. The two companies are quite different on the surface, but Young explained that they are both attempts to do the same thing: give customers control over what they want to do. When Red Hat was competing for customers with Microsoft, he said, Microsoft was totally unwilling to offer customers the control that Red Hat was, so Red Hat gained a lot of contracts and grew into an enormous enterprise of its own.

Lulu is different, he noted, in the sense that the competition—most notably Amazon—has seen Lulu's business model and responded by offering its own, similar service. Thus, Lulu has a different kind of battle to fight. It is still a really fun project, he said, but time will tell whether or not what it offers to authors is significantly better than Amazon's pitch, only marginally better, or if Amazon comes up with something better.

Several audience members asked about how the community should respond to resistance toward software-freedom issues. One asked specifically how to communicate free-software ideals to politicians; another asked about businesses' resistance toward the GPL. On the political front, Young said that there are a number of players today who take the correct approach (Lawrence Lessig, the EFF, and Public Knowledge, for example). They emphasize transparency, which is important to politicians. But Young also advised everyone listening to understand that they, too, have a role to play in such conversations. If you ever meet a politician, he said, make sure the politician knows what you consider important. Money in politics is a problem, but all politicians ultimately care more about votes than money.

As for businesses' resistance toward the GPL, Young said that it would be better to ask Moglen or Karen Sandler for the real details. But he noted that in the early days, Stallman had considered putting his software into the public domain. That would not work, Stallman concluded, since the "public domain" is not a concept that is protected against abuse (in this case, against turning a public-domain program into a proprietary one). The GPL and the concept of "copyleft" that it implements, he said, create the legal protection for the public domain. There are other licenses, but there is no more effective tool for the "real" public domain. And that real public domain is the thing that gives users (or customers) control, which, in turn, is the one thing proprietary software cannot offer.

"Control" as Young articulated it in his talk is, essentially, the same idea as the freedom that has always been central to free software. But it is interesting to hear it described in such different terms. Young jokingly called himself "an evil capitalist" several times, and perhaps control is terminology that fits better into vendor-to-customer conversations and sales pitches than does freedom. But Red Hat's success over the years certainly demonstrates that Young's way of explaining the issues is one that resonates with quite a few users.

[The author would like to thank LCA 2015 for travel assistance to Auckland.]

Comments (9 posted)

Software-defined radio at the OpenRadio miniconf

The first two days of linux.conf.au (LCA) are reserved for a series of miniconfs proposed and run by the conference delegates. This year, one of Monday's miniconfs centered around free-software and amateur (ham) radio, especially where it involves software-defined radio (SDR). The ham and free-software fields have a great deal in common—especially at the philosophical level—but, practically speaking, using free software with SDR can be a bit clunky. The OpenRadio miniconf attempted to kickstart interest in the topic, both by including relevant presentations and by giving attendees the opportunity to construct their own device: a new SDR peripheral that is more flexible and more open than most of the competition.

Sessions

The miniconf day started and ended with plenary talks. Don Wallace, who is the Overseas Liaison Officer for the New Zealand Association of Radio Transmitters (NZART, New Zealand's amateur radio association) gave an overview of the radio-spectrum allocation process and how New Zealand's allocation rules fit in with the multi-tier regional and global system of radio regulation. He also discussed the current landscape of ham radio allocations. Notably, he pointed out that almost all existing frequency allocations are at risk, as more and more spectrum is being allocated for cellular telephone systems every year.

He also explained the background of the unlicensed industrial, scientific, and medical (ISM) bands, where the miniconf's SDR peripheral is designed to operate. The 2.5GHz and 5GHz ISM bands were initially designed for use in operations like welding, microwave oven heating, and medical devices. It was only co-opted for use in wireless networking when Cisco and other networking hardware manufacturers balked at the costs that would accompany a new spectrum allocation. This strange pairing continues to be a problem for ham radio and for wireless networking as new devices come to market: the cumulative RF noise generated by the radio sources in compact-fluorescent and LED bulbs is rapidly reaching a problematic level. During a recent city-wide blackout, he said, the RF noise floor dropped by 20dB. It can be hard to work in such a noisy radio band, he said, and ham radio operators are often perceived as the least important group when national and international regulations are being re-examined.

Next, Paul Warren gave a talk (the slides from which are available online [ODP]) comparing and contrasting the current crop of "open" ham radio transceivers. There were a great many options covered, from stand-alone hardware devices that take microphone input and produce audio output (which makes those devices the most similar to traditional ham radio gear) to peripherals that require a computer for their functionality. The sense in which each device is "open" varies considerably as well, he explained. Some of the devices include schematics and printed circuit board (PCB) designs—which makes them "open hardware"—but they might use firmware that is not released under a free software license (if its source is published at all). Others might be the reverse: fully free software, but without plans or schematics. In selecting which devices qualified for inclusion in the talk, Warren sad, he just made his best guess for which projects "were attempting to do the right thing."

A thorough examination of the options is important for anyone considering a purchase. Warren identified a particular favorite device at the moment in the PortableSDR, a powerful but handheld-sized device that recently won third prize in a project-building competition held by the Hackaday web site. He also looked at several devices he called "SDR exciters" rather than true transceivers. These include well-publicized projects like the HackRF and BladeRF. Some of them are quite nice, he said, but they belong in a different class because they come with serious limitations: very low transmission power and a lack of filtering on input signals. Due to these limitations, most are good for experimentation, but would require modification to do any long-range or sensitive communication.

Last but certainly not least, Paul Campbell gave an overview of his work designing "smaller, cheaper, open wireless" for Internet-connected devices. The project is still in the experimental stage, but more details can be found at his web site Moonbase Otago. The project in question is a type of extremely low-power mesh-networking device. The devices communicate with the IEEE 802.15.4 low-power wireless Personal Area Networking (WPAN) networking layer. 802.15.4 is best known for ZigBee products, although ZigBee communication is actually implemented in a higher-level protocol on top of 802.15.4.

Moonbase Otago's cheap RF devices are built around a tiny CC2533 CPU and very few physical parts: just a handful of inexpensive resistors and capacitors. The CC2533 provides serial communication, I2C, and several GPIOs; just 32K of storage is available, plus 4K of RAM and five timers—but there is a AES encryption chip. Campbell's team has developed a tiny operating system (occupying 6K) that provides threading and task queueing for applications, but also incorporates several interesting features. Using the AES chip, all communication is encrypted automatically: paired devices receive messages from each other, and drop any that use a different key ID in the message packet. The devices can also update application software over the air, propagating updates from device to device automatically.

The project is still in active development. Because GCC does not support the CC2533 CPU, the team has been using the free-software SDCC compiler, and has had to reverse-engineer a number of features of the device where the manufacturer's sample code does not work. Writing a GUI development tool is underway as well, and the boards will support a variety of Arduino-compatible macros to assist new developers.

Hardware

The majority of the OpenRadio miniconf schedule, however, was devoted to the construction of the OpenRadio SDR device. The board is a radio transceiver, designed by Mark Jessop. The basic architecture is modeled on the open-hardware SoftRock SDR board, with some modifications intended to reduce cost and a few enhancements to its functionality. Kits were manufactured ahead of LCA, and attendees purchased them online to pick up at the beginning of the workshop session.

An Arduino Nano is used to control the board's local oscillator, setting the receive frequency, over I2C. The Nano also does phase-shift-keying (PSK) modulation and generates transmitter output. Incoming signals are mixed with the local oscillator signal to produce a baseband audio signal that can be fed into a host computer via any sound card with a stereo input.

The board is designed to run on the 27MHz ISM radio band, although it can be used with other bands. Another of the distinctions between the OpenRadio and the SoftRock design on which it is built is that the OpenRadio hardware includes a prototyping area: a set of large pads onto which the user can construct a set of filters to change the operating frequency. The SoftRock can also be constructed to tune to one of several specific radio bands, but the OpenRadio has a pluggable option, too: the frequency filters can be built on removable daughterboards (instead of being soldered directly to the prototyping area) so users can make several filters and swap them out as desired.

The Arduino code that runs on the Nano is available on GitHub. Several applications are available to run on the host computer, to read samples from the stereo input and process the signal. Both the Python-based Quisk and fldigi were in use around the room once testing began. Both programs can demodulate the radio signals received by the board, decoding a wide variety of signals modes—from single side-band (SSB) voice to digital data packets of various varieties.

Unfortunately, even given the simplicity of the kits, time did run fairly short by the end of the day. Not everyone was able to complete the assembly process, although quite a few did. Afterward, OpenRadio miniconf organizer Kim Hawtin said that he hopes the kits will be just the first step, and that the project will be able to build on the initial design at subsequent events.

Ham radio has long been a popular hobby among free-software fans, so it can come as an understandable surprise to hear supporters of both movements say that there is important work left to be done in a popular area like SDR. As the LCA OpenRadio miniconf demonstrated, though, even when there is considerable interest, the hardware is far from standardized (even on items like licensing) and the free-software tools for working with SDR tend to be of the "toolkit" variety rather than the full-blown application variety. But the miniconf sold out in advance (there were a fixed number of spaces available in the workshop); one would hope that this and the increasing number of projects available are indicators that the community is actively maturing and good things are still to come.

[The author would like to thank LCA 2015 for travel assistance to Auckland.]

Comments (none posted)