Leading items

Replacing Google with microG

A common criticism of free-software projects built for Android is that they all too often rely not just on the frameworks and libraries that are part of the official Android Open Source Project (AOSP), but on the proprietary APIs implemented in various add-ons from Google—such as the Google Maps API or the Google Cloud Messaging message-broker service. Working around these Google-supplied components is not trivial, but there is at least one effort underway to provide a drop-in free-software replacement: microG.

Google's proprietary Android service layer is currently called Google Play Services (although it went by other names in prior releases). It provides around a dozen APIs that link to online services run by Google; they include the Google Maps service, the Google Drive storage service, a network-based geolocation API, the Google Cast screen-sharing API, a social/sharing API for mobile games, and hooks for various services like Google+, Google Analytics, and Google Wallet. Google Play Services comes pre-installed in essentially all Android devices sold by major vendors or mobile carriers.

The importance of these APIs depends a great deal on what one wants to do. Certainly it is to be expected that Google's own Android apps will make use of the APIs. K-9 Mail does not depend on Google Play Services but Gmail, to no one's surprise, does. Where matters begin to get fuzzier is with apps like the Google Play Store app store itself: that app depends on Google Play Services, even if it is only used to download and install self-contained apps. While there are alternative app-installation and app-management mechanisms (F-Droid being the most familiar to the free-software community), they contain only a fraction of the main store's contents.

Where the issue peaks, however, is with Android apps that are, themselves, free software. Users wishing to have a fully free-software system (or to simply avoid Google products) can install AOSP alternative base systems like Replicant or CyanogenMod and either install apps directly from .apk packages or via F-Droid, but if a particular app uses one of the Google Play Services APIs, the device inherits that dependency in spite of all the groundwork. The pain felt by the would-be Google avoider is most keen where privacy- or anonymity-centric apps like Signal are concerned, since the APIs provide the potential for Google to fingerprint or track the device. At the very least, removing a dependency on Google Play Services is frequently highlighted as a feature in projects like Silent Circle's Blackphone and porting efforts like LibreSignal.

Free to the core

In general, there are two possible approaches to solving this dilemma: alter each app to replace Google Play Services calls with non-Google equivalents, or replace the Google Play Services framework itself. There are various efforts to patch out Google Play Services dependencies for individual free-software apps, but those efforts are, by their very nature, single-app solutions at best.

microG is an effort to take the latter approach instead. It grew out of an earlier project called NOGAPPS, started in 2012, that focused initially on the Maps and network-geolocation APIs. Early builds replaced Maps functionality with OpenStreetMap and the network-geolocation service with the Mozilla Location Service (MLS).

microG has since migrated to GitHub and expanded its scope, with the goal now stated as providing a "free-as-in-freedom re-implementation of Google's proprietary Android user space apps and libraries". The project's base library package is called GmsCore, which is designed to eventually serve as a full replacement for the Google Play Services library.

GmsCore currently supports the Cloud Messaging and network-geolocation APIs fully, and provides partial support for the Google Authentication API, Maps API, and Google+ API. That said, GmsCore is also known to cause crashes (rather than simple lack-of-functionality) for apps expecting the Google Auto and Google Cast APIs. The project also provides a stand-alone library package for the network-geolocation service (UnifiedNlp) and packages for two older APIs: the now-deprecated version one of the Maps API (mapsv1) and a stand-alone version of Google's Cloud Messaging library (GsfProxy).

In the past, the project had worked on a free-software reimplementation of the Play Store app itself, called BlankStore. That effort is no longer active, however, the project does provide a similarly named app called FakeStore. The purpose of FakeStore, though, is to fool other apps into believing that the authentic Play Store app is installed. Some apps will refuse to run otherwise. FakeStore, however, provides no other functionality.

At the moment, getting started with microG requires a fair amount of serious groundwork. First, only Android ROMs that support spoofing package signatures are currently supported. This is a bypass for the Android security feature that verifies the owner of the key used to sign packages. Spoofing allows GmsCore to pretend to be Google's actual Google Play Services library and FakeStore to pretend to be the official Play Store app, when other apps check for the presence of those specific packages. There are a few third-party Android ROMs that ship with that feature enabled; others will need to install a special package, patch their ROM, or build a custom Android ROM for their device.

Next, the real Google Play Services package must be removed if it was installed (and, with it, all of its dependent packages). Users can then install GmsCore, GsfProxy, and either BlankStore, FakeStore, or the real Play Store app.

Looking at the GmsCore wiki, one might think that only a few apps are supported by the limited functionality currently available: it notes only Signal, Play Music, and YouTube as known to work with the Cloud Messaging API. But the issue tracker tells a different story; users have reported quite a few functioning apps, including other popular offerings like Uber and Lyft, as well as various free-software apps like Matrix Console.

Mind the GApp

What any judgment on microG's viability comes down to, it seems, is an assessment of what constitutes an "important" app or API. No two users may ever agree on exactly where to draw that line. But, on that front, it is also important to remember that Google Maps is often cited as the "killer app" for Android. At the very least, many end users seem to find mapping and geolocation to be a critical feature: many other smartphone services boil down to little more than good network connectivity for general-purpose communication, but tying the device into its real-world location is a far more useful capability than Google Wallet's micropayments or any social-sharing feature found in a game.

Thus, it is undoubtedly a wise move for the microG project to focus first on the mapping and geolocation features. The project's team is still rather small: there are just 14 contributors to GsmCore, with founder Marvin W responsible for 85% of the commits. As the project's work gains more functionality and exposure, it will surely attract more contributors, although it will be a difficult battle to keep pace with the changes introduced in regular Google Play Services updates.

Perhaps the best chance microG has for widespread success would be to join forces with other like-minded free-software Android projects. The CyanogenMod community, for instance, has for years relied heavily on the quasi-legal redistribution of Google Android apps through projects like OpenGApps (which, despite the name, distributes only binary packages of the official Google apps). Having a real, free-software implementation of the needed libraries would be a boon to users and developers alike.

The Replicant and Blackphone communities would also likely find microG a welcome addition, removing a dependency on a large, proprietary blob and replacing it with a free-software package. Even further out, the fact that Google uses its Google Play Services framework as a (rather large) bargaining chip to keep Android vendors in line likely rankles some within the industry; losing access to the framework would mean a vendor must ship devices incompatible with many popular apps. Any project removing the power that Google Play Services holds over device-makers is surely valuable. Then again, microG has done quite well for itself so far; perhaps it has no need to change its game plan any time soon.

Comments (6 posted)

KubeCon part 2: 1.3 and the CNCF

Having covered the major features of Kubernetes 1.2 (as described in the first part of this series), speakers at KubeCon.EU moved on to talking about the future. With the project's rapid development, version 1.3 isn't that far in the future. Also, contributors were very interested in discussing Kubernetes' new membership in the Cloud Native Computing Foundation.

Community and Kubernetes 1.3

In the "State of The Union", David Aronchik, Kubenetes Project Manager for Google, brought folks up to date with what's happened in the Kubernetes community and what's ahead for version 1.3 and beyond.

By any of several measures, the Kubernetes community has been growing rapidly. The project has over 25,000 commits, representing a steady increase for each version; that puts it in the top 0.1% of projects on GitHub. Over 680 people contributed to version 1.2, about a third more than version 1.1. The community has created 85 Meetup groups worldwide. He cited several reasons for this accelerating development: a well-designed, scalable platform; a responsive and active community; and a big name and good publicity. "Who wouldn't want to run like Google?" he asked.

According to analyst firm RedMonk, however, the vast majority of contributions still come from Google and Red Hat. If RedMonk's analysis is accurate, the number of new contributions is primarily evidence of increased commitment on the part of those two companies than of anything else. However, the increase in the number of meetups and other online traffic may show that the user base is growing.

Aronchick mentioned some of the 1.2 features not covered by other speakers. There's a new GUI, called Kubernetes Dashboard. The team has made Kubernetes scale to large numbers of nodes and pods, from around 200 nodes and 5000 pods to around 1000 nodes and 30,000 pods. They've also added a lot more API endpoints so that users can swap out components of Kubernetes, such as the DNS or scheduler, for their own components.

Kubernetes 1.3 is expected around July — somewhere around 16 weeks away at the time of Aronchick's speech. It's still under very heavy development, and he urged attendees to participate in the weekly community meeting. He described some of the features they're working on in broad outlines.

Contributors are working on cluster federation, nicknamed "Ubernetes." Right now, each Kubernetes cluster is a single unit that is expected to run in one data center. Users who want geographic distribution for high availability purposes have created several hackish workarounds to enable it. The new feature would allow users to see clusters in multiple data centers as one continuous infrastructure, and schedule jobs across the distributed cluster as if they were all local. In turn, this would let some companies replace Amazon Web Services with a Kubernetes-based architecture.

The Scheduled Jobs feature has already been merged and will be released in version 1.3. It enables running containers to execute scheduled tasks in the cluster, much as the cron utility does for a single server. Aronchick also expects in-cluster Identity Access Management (IAM) in 1.3., but did not have many specifics on this.

Kubernetes Joins the CNCF

Aronchick ended his talk by announcing that Kubernetes had joined the Cloud Native Computing Foundation (CNCF). He explained that the entire codebase for Kubernetes had been given to the CNCF. Google didn't create Kubernetes as a business, he explained, "we want to enable other people's businesses."

The CNCF was launched last July with the management and organizational support of the Linux Foundation. Its list of founders is a veritable "who's who" of the Linux container ecosystem, including Google, Docker, CoreOS, CloudFoundry, Mesosphere, and many others. The organization's goal is to advance the technology for "cloud-native" applications, which are defined as services that are packaged as containers, dynamically managed, and micro-services oriented. Despite this broad corporate support, the CNCF didn't have any open source projects to oversee until KubeCon. With the many ties between Kubernetes and other projects, we may see the CNCF having a more active role in the world of open source now.

Over the last eight months, the CNCF has been developing a governance system. After the sponsors appointed a board, that board approved a charter. One of the things established in this charter was the Technical Oversight Committee (TOC), whose members were first announced at KubeCon.EU. The TOC is a supervisory committee which will oversee the open source projects that join the foundation.

Aronchick introduced Alexis Richardson of Weaveworks, who is the new chair of the TOC. Richardson explained that the mission of the organization "is to be the Apache of cloud-native." By that, he means that it should become the legal and financial resource for all projects that fit their definition of "cloud-native". "All of our projects have the same goal, to make your infrastructure scale faster through automation."

Richardson pointed out Google had done a fairly unusual thing with Kubernetes. It had rewritten an internal tool for new container technology, and then open-sourced it. He said that the motivation for Kubernetes joining the CNCF was to encourage contributions. Many potential contributors didn't want to sign a Contributor License Agreement (CLA) with Google, for a variety of reasons. Now that the code belongs to a non-profit more people should be able to contribute.

A panel of community leaders later in the day discussed what joining the CNCF means for management of the project. An audience member asked what paperwork the CNCF would require from contributors now. They explained that for employees of supporting companies this is simple because it's covered in their membership. For other contributors, the foundation has yet to determine how things will work. It may require a CLA, but at least one TOC member is hoping that they can use a Developer Certificate of Origin (DCO) instead (as Linux does) since this is less of a barrier to contributions.

According to the panel, the CLAs and similar paperwork are the only thing that should change; their intention is to handle the "legal stuff" and not govern the project. Speakers contrasted the CNCF's more hands-off approach with the close stewardship of projects by the Apache Foundation. "The more you become a bureaucracy, the more you attract bureaucrats," said one.

There are areas where the CNCF might provide guidance, such as the selection of new committers. Right now there's no process at all to decide who becomes a committer in Kubernetes. They also plan to copy at least one Apache practice, that of having an "incubator" for new projects joining the foundation. That means that Kubernetes is technically an "Incubator Project" right now, although that may be more for the benefit of the CNCF than the project.

Conclusion

KubeCon.EU 2016 also included many more speakers from several different organizations, showing how the young technology has already fostered diverse solutions. Eric Lewis of the New York Times explained how they use Kubernetes and containers to manage their complex web presence. Appsembler staff talked about scaling out Massive Open Online Courses (MOOCs) using it. Jacob Tomlison of The Met Office went over how they implement Kubernetes auto-scaling in order to model weather and climate. Matthew Garrett explained how Trusted Platform Module support, recently added to CoreOS, can be used to implement admission control for container services.

Kubernetes is rapidly growing in every direction and seems to be destined for a bigger, more exciting future. We can expect frequent developments both from the project, and in the competition between container system orchestrators. Regardless of what happens, LWN will cover it.

[Josh Berkus works for Red Hat.]

Comments (none posted)

A single Node of failure

The web-development community was briefly thrown into chaos in late March when a lone Node.js developer suddenly unpublished a short but widely used package from the Node Package Manager (npm) repository. The events leading up to that developer's withdrawal are controversial in their own right, but the chaotic effects raise even more serious questions for the Node.js and npm user communities.

npm itself is a module repository for Node.js code, akin to the Python Package Index or similar repositories for other languages and frameworks. Users can install a package with a simple npm install foo, but the service is also widely used by Node.js developers to automatically fetch and install dependencies: projects list their dependencies in the package.json file, and they are recursively fetched from npm and installed when the package is built. Using npm in this manner is standard operating procedure, allowing complex JavaScript applications to be written on top of multiple third-party frameworks in minimal lines of code. The service, however, is run by a private company called npm, Inc., rather than by the Node.js project.

Kik starter

The trouble began with Azer Koçulu's module kik, a command-line tool designed to ask a few simple questions and set up a new Node.js project template based on the answers. According to Koçulu, a lawyer representing Kik Interactive (makers of the Kik instant-messaging app) contacted him by email and insisted he change the name of the package. When he refused, the company took its demand to npm, Inc. instead, copying Koçulu on the emails. npm founder Isaac Z. Schlueter then took ownership of the kik module on npm from Koçulu and unpublished it, without Koçulu's consent.

That action, it seems, frustrated Koçulu to the point where he decided to pull all of his other packages from npm as well; in his post on the subject, he commented that the situation "made me realize NPM is someone’s private land where corporate is more powerful than the people" and "NPM is no longer a place that I’ll share my open source work at". He concluded by apologizing for any effect the move might have downstream:

The story might have ended there, except that Koçulu was the owner of quite a few modules on npm. And, as luck would have it, one in particular, called left-pad, was part of the dependency chain for a host of high-profile Node.js projects and frameworks—including React.js, Babel, and Ember.js. When left-pad disappeared from npm, it caused a chain reaction of dependency failures that broke scores of JavaScript project builds.

Although no major interruptions appear to have hit live services, the widespread breakage of automated builds caused considerable panic; Daphne Maddox now famously described the impact as "This kind of just broke the internet." Reacting to the outcry, npm's Laurie Voss restored the unpublished left-pad module to the repository. On Twitter, he commented that "Un-un-publishing is an unprecedented action that we're taking given the severity and widespread nature of breakage, and isn't done lightly." Later, he said that Koçulu's decision to pull left-pad from the repository "puts the wider interests of the community of npm users at odds with the wishes of one author" and that the restoration was made because "I cannot see hundreds of builds failing every second and not fix it."

npm later posted its account of the events, citing the service's package name dispute resolution policy as the rationale for removing Koçulu as the owner of the kik module name. It also defended the re-publishing of Koçulu's left-pad module, noting that most of the downstream builds inherited their left-pad dependency through the line-numbers package, which was hard-coded to depend on left-pad 0.0.3. Even though another developer (Cameron Westland) created a new left-pad module shortly after the original was pulled, the new module was numbered version 1.0.0, which did not fix most of the dependencies.

For its part, Kik also posted its side of the story, quoting from the emails exchanged between Koçulu and the company representative, who it notes is the company's patent agent and not a lawyer. The post expresses regret for the unfolding drama:

Interestingly enough, Kik's representative evidently told Koçulu that the company would "have no choice" but to take down his accounts "because you have to enforce trademarks or you lose them." That is a common enough refrain, although as attorney Pamela Chestek recently pointed out, the "defend it or lose it" idea is not actually part of US trademark law. How one interprets the email exchanges between Kik and Koçulu will, no doubt, vary. It does seem, however, that Kik sought a compromise of some sort; Koçulu quoted a buyout price, which Kik did not accept.

Although many who commented on the string of events were clearly upset at Koçulu, he had his share of supporters as well, in light of the naming dispute with Kik. Tom McGee, for instance, wrote on GitHub: "Forget my broken build, stick it to the man!!" The past week has also seen a flurry of new protest projects appear on npm, using the name kik or otherwise referencing left-pad.

Left behind

As to the widespread cascade of disruption caused by the missing left-pad module, the community voiced several concerns about how npm is managed. First, providing the ability to unpublish a module is controversial. In its blog post, npm cited "historical reasons" for retaining the feature, but many other package repositories simply do not make it possible to remove an existing release on which other code depends. Mozilla's E. Dunham, for instance, noted that the Crates.io repository used by Rust projects allows developers to "yank" a release, which prevents it from being added to new builds, but does not break existing dependency relationships.

Furthermore, npm's package-publishing rules created yet another bind in that, once pulled, no other developer could publish another left-pad module utilizing an already-used version number. Thus, even though Westland published his replacement left-pad module just a few minutes after the Koçulu unpublished the original, it could not satisfy the hard-coded version dependency in line-numbers. That limitation is what forced Voss to take the "unprecedented action" of undoing Koçulu's withdrawal of the 0.0.3 release.

As several of the initial commenters on GitHub bug reports noted, another problem with npm is that modules are not namespaced by origin or user account. That made it impossible for downstream projects to simply switch from Koçulu's left-pad package to Westland's.

But perhaps the broadest issue raised is that npm itself is not managed by the Node.js community. Instead, it is run by a private entity and not subject to oversight. Koçulu's departure stemmed directly from what he saw as npm management siding with a for-profit company to take away his access to his own work. Many of the supportive comments from other developers seemed to agree—either expressing discontent with npm's unilateral action, or concern that no agreed-upon policy seemed to be at work.

There are, after all, many existing modules available in npm that use names trademarked by someone in the software industry, from well-known brands like Facebook to more generic words like square. The developer behind the newly minted not-kik package even challenged npm on the point, describing it with: "It's not kik, and never will be. But it has kik in its name. What now, NPM Inc.?"

Bryan Cantrill suggested that the "inescapable conclusion" was that the npm registry should be managed by the Node.js Foundation. So far, that idea does not seem to have gained much traction.

Lazyweb

The other side to the debate is concern that such a simple module as left-pad was widely enough used in the first place to wreak so much havoc when it disappeared. The module was eleven lines long, and served only to left-pad strings. Surely, the argument goes, such a simplistic function is too small to demand its own package—or, better yet, its purpose is so simple that any developers worth their salt should have implemented it themselves in mere minutes.

David Haney provided an in-depth critique of the Node.js community on precisely those grounds, calling the left-pad catastrophe symptomatic of a "stringing APIs together and calling it programming" mindset in the npm ecosystem, with developers "writing the smallest amount of code possible to string existing library calls together in order to create something new that functions uniquely for their personal or business need."

He added: "What concerns me here is that so many packages took on a dependency for a simple left padding string function, rather than taking 2 minutes to write such a basic function themselves." While the sentiment was echoed elsewhere on various discussion forums, Node.js developers as a whole, it seems, did not take the criticism lightly. The second comment on Haney's post chided him for "fundamentally misunderstanding the npm ecosystem and small module philosophy". Whether that refutes or underscores Haney's argument seems to be largely a matter of perspective.

Apart from the effort it takes to write a string-padding function, many people have pointed out that the hard-coded version dependency in the line-numbers module is bad form in its own right. Furthermore, even when it was unclear whether or not left-pad 0.0.3 could be restored to the npm registry, many developers complaining about broken builds seemed unaware of the fact that there are other ways (outside of npm) to specify a dependency in a Node.js package. One could even have pointed directly at Koçulu's left-pad code on GitHub.

Node War II?

On March 29, npm announced that it would, indeed, be changing its module-unpublishing policy. In the new policy, authors have a 24 hour window in which a published version can be unpublished. After 24 hours, authors will have to contact npm's support team, who will check whether or not the removal will break any dependencies for other packages. Furthermore, if every published version of a module is unpublished, npm, Inc. will take over the package name to prevent "malicious squatting". The post calls the new policy "a first step towards balancing the rights of individual publishers with npm’s responsibility to maintain the social cohesion of the open source community". Time will tell how well it will be received by the Node.js community.

The kik/left-pad debacle hinged largely on version numbering and package naming, but the incident is far from the only criticism leveled at npm. For instance, Google's Sam Saccone found that npm module installation poses a significant security risk. The npm command-line tool allows downloaded packages to run scripts with the full privileges as the current user; this means that a malicious script hidden in a dependency might run arbitrary commands as a user (or even as root). Among the possibilities is that such a script could publish updates of the user's modules directly to npm, allowing it to spread as a worm.

Others, like Armin Ronacher, warn that npm's "micro-dependency" approach leaves open several other possible exploits, starting with a "dependency explosion" that creates more packages than a reasonable person can be expected to maintain with scrutiny. One individual developer, he noted, maintains 827 separate modules on npm. Even if they are small, he said, "it's significantly harder for him to ensure that all of those are actually his releases". As recent events have shown, even a small module can be the source of considerable trouble. The open question is whether or not npm—and the larger Node.js development community—will learn from this experience and be able to self-correct.

Comments (56 posted)