Leading items

Mozilla and Pocket

Starting in version 38.0.5, Firefox includes a built-in integration with the bookmarking service Pocket. Although the Pocket service has been available in Firefox through an extension for several years, the integrated feature sparked an outcry among some users. Critics raised a variety of perceived problems with the feature, but most of the backlash focused on the proprietary nature of the Pocket service or on the perception that the feature resulted from a secret deal between the company and Mozilla—a deal that, presumably, did not take the community's best interests into account.

Recent history teaches that Mozilla should probably expect blowback whenever it adds a Firefox feature that involves cooperation with a closed-source service or company—implementing the W3C Encrypted Media Extension (EME) API or H.264 support, for example. Though blowback should perhaps be expected for every new Firefox feature (see the controversy about signed extensions, for example). In any case, although the past week has seen a rise in public debate about the Pocket feature (with blog posts critical of Mozilla from Benjamin Kerensa and Julien Voisin, among others), the feature itself is more than a month old, which warrants examining it in its historical context.

The Firefox 38.0.5 release landed on June 2. Pocket integration adds a button to the toolbar; clicking on it essentially allows the user to store the URL in a per-user Pocket account, from which it can be looked up and read later. In that sense, Pocket is no different than a traditional bookmark, except that a user's Pocket list is accessible from non-Firefox browsers (unlike bookmarks synchronized with Firefox Sync).

The addition of the feature was mentioned in the release notes and accompanying blog post, but some users seemed to find that degree of communication insufficient. For one thing, the 38.0.5 release is a "point point" release, which is not the normal place one expects to find the introduction of a significant new feature. For another, the feature evidently landed for Firefox 38 without first spending the usual amount of time in the Nightly channel—which, again, is the expected behavior. Many users—including Nightly channel testers—were taken by surprise when the feature appeared.

Questions

The most detailed critique of the feature, though, took place on the Mozilla Governance mailing list. Tucker McKnight filed a bug report about the move, in which he listed several issues. Shortly thereafter, McKnight was told to take the topic to the mailing list instead—which he did, there reiterating his concerns. McKnight focused on implementation details, starting with the fact that the Pocket integration is not implemented as a Firefox extension, but as native code. This, he said, raises three concerns:

• Extensions can be removed entirely, but Pocket support can only be disabled.
• Pocket support can only be disabled through the about:config page, which is not user friendly, "and therefore not in line with Mozilla's mission. In the past, Mozilla has been very good about showing the user what new features have been added to the interface and explaining any privacy implications that may come with them."
• Pocket support uses the user's existing Firefox Account to sign in to the Pocket web site. "It may also not be clear to some users that, even when signing in with your Firefox account, you are still giving your email address to a third party whose privacy policy is different than Mozilla's."

Adam Porter replied, raising the lack-of-public-discussion issue, and also pointed out that the move gives favored status to a proprietary service at the expense of similar free-software projects (like wallabag). A more appropriate approach, he said, would have been to define a "save for later" API that Pocket and other projects could hook into.

The ensuing back-and-forth was, at times, overly heated—in ways that will sound familiar to those experienced in Internet discourse. A number of community members chimed in just to express outrage and announce that they were switching to Chrome, and some Mozilla employees lashed out at the critics to accuse them of being uninformed.

If one takes away the emotion, though, a few key points remain. Some critics objected to the Pocket feature because Mozilla has historically resisted adding functionality to the core Firefox code that could easily be implemented in extensions (ad blocking, for example). That philosophy was one of the original justifications for decoupling Firefox from the Netscape suite, so changing it now seems like a policy shift. Similarly, others pointed out that "back in the day, Mozilla implemented Mozilla Weave (now Firefox Sync) exactly because existing alternatives were proprietary." Thus, partnering with a proprietary vendor is an about-face, one that is particularly noticeable given that Mozilla dropped its Pocket-like "Reading List" feature at the same time.

Finally, a few critics raised specific objections to the privacy policy and terms of service (TOS) for Pocket. At the very least, the language of both documents is written to apply to an installable software project (as the Pocket extension was), while the new Pocket-integration feature is implemented as a set of web API calls. Those API calls use a pocket namespace, which adds some additional fuel to the argument that the feature favors one vendor to the exclusion of all others.

Most critics seemed to feel that Pocket, as a commercial entity, should not be implicitly trusted with user data, and many worried that the privacy policy allows Pocket to change its mind and begin commercializing the submitted data—leaving little recourse to users. Others raised concerns about the US-centric language in the policies and about prohibitions on using the service commercially or with objectionable (to some) links.

Answers

For its part, Mozilla representatives have provided responses to most of the core criticisms. Gijs Kruitbosch, a Mozilla engineer who worked on Pocket feature, answered both the lack-of-discussion and "playing favorites" criticisms. The feature landed late in the development cycle, he said, so the API and preference names were written specific to Pocket for the sake of speed—but the plan is to generalize them in future releases. Furthermore, Mozilla is using the Pocket implementation to gather usage data that will lead to a more open API once the use patterns and requirements are better understood. Mozilla's Mike Connor added that the same approach was taken for the first versions of search-engine integration and Firefox's Social API.

As to the concern that Pocket is a closed-source service, Mozilla's Gervase Markham replied Mozilla has partnered with closed-back-end services in the past without raising ire—most notably "the bundled search engines, safe browsing and (until recently) our location service". He did, however, agree that the UI's perceived ambiguity about the fact that user data is being sent to a third party is a valid complaint.

Ultimately, though, Mozilla could not provide easy answers to every question—in particular, to the privacy and TOS concerns. Dan Stillman called the comparison to search-engine integration invalid, given that Firefox already had a bookmark-sync feature that did offer privacy safeguards:

Connor noted that Mozilla's bookmark-saving web service, Firefox Sync, was designed with strong cryptography and strong privacy protections in mind, and that it failed to catch on. "The vast majority of users didn't understand or care about the added security. It was more of a liability than an asset. Firefox Accounts make a different tradeoff as a result, and it's unsurprisingly more popular (and _useful_) as a result." Meanwhile, he said, Pocket has already proven itself popular—both as a browser extension and on other platforms (such as e-readers and mobile OSes without Firefox).

On June 10, Markham volunteered to get clarification on the Pocket TOS and privacy policy as they apply to the Firefox-integration feature. On July 14, Urmika Devi from the Mozilla legal team joined the discussion and gave a blanket answer to the policy questions:

It remains to be seen how Devi's response (which also addressed some of the specific, recurring concerns) will be interpreted, but the legal team has agreed to follow up on any additional questions.

Nevertheless, there remain other unanswered questions, too. For example, Stillman, McKnight, and several others requested more information (and even a timeline) about when and how the "save for later" feature now used only by Pocket would be opened up to additional participants, as Kruitbosch suggested it would. Others have asked whether or not the Pocket deal provides revenue to Mozilla. There has not yet been a reply on either point. Whatever else Mozilla may have in mind for the feature, this debate indicates that one thing it certainly needs is improved clarity and communication with the community.

Comments (15 posted)

A split within the Tox project

Key developers from the Tox encrypted-chat project recently parted ways with the project's titular leader, after a dispute over the misuse of donated funds. The team has set up a new web site and source-code repository, but the old sites remain in place, too. In addition, the conflict over the project's funds could have a serious impact on Tox's Google Summer of Code (GSoC) projects—not to mention the project's future.

The root of the disagreement is an accusation that Sean Qureshi, the sole board member of the Tox Foundation, withdrew funds from the Tox Foundation's bank account for personal use, and has refused subsequent calls to rectify the situation. How the Tox project collects and manages donations has, evidently, been a point of contention for quite some time—prompting other developers to depart. But this incident appears to have been the breaking point, which led to an acrimonious split between the parties.

The Tox project was started in mid-2013 by users on the 4chan message board. At its heart, it provides a peer-to-peer encrypted messaging service. The main project, however, develops only the core library and protocol; outside contributors are responsible for all of the client software. Qureshi has long acted as project leader (maintaining the domain registrations, managing infrastructure, coordination activities, and so on) but has never been lead developer—that is an individual who chooses to go only by "irungentoo" online.

In June of 2015, rumors began to circulate that Qureshi had withdrawn funds from the Tox Foundation bank account and used them on personal expenses. Those funds included both individual donations and money collected for mentoring GSoC students. As word of these events spread, several contributors to the project announced their departure, criticizing Qureshi as well as irungentoo—the latter for not taking action.

On July 5, GitHub user "rr44" opened up a bug against Tox asking for an explanation. Irungentoo replied, saying:

On July 11, he followed up by re-launching the Tox project at a new site: tox.chat, rather than the previous tox.im, and by making an announcement on the new site's blog. There, irungentoo described the events in more detail, saying that Qureshi admitted he "'took a loan against the Tox Foundation', and used the entirety of the foundation’s funds on personal expenses completely unrelated to the project." Irungentoo issued an apology to the community, saying "we certainly could have handled our finances in a more responsible and transparent manner" and "we can blame no one but ourselves for this."

He also said that one reason for the lengthy delay between the first rumors of misconduct and the formal split was that project members had spent a lot of time trying to privately rectify matters with Qureshi. In the end, though, no such reconciliation was achieved, so the development team felt forced to disassociate itself from Qureshi and the Tox Foundation, and begin again with new servers and new domains. As for finances, the announcement said only that "we will not be taking any official donations until we have set up a proper organization with an emphasis on transparency and protection of assets (more details on this at a future date)". It also noted, though, that many developers would accept individual donations.

The nebulous status of the Tox Foundation seems to play a major role in the story. Several commenters, including rr44, complained that the Tox Foundation, despite its name, was not set up as a non-profit charity. Unfortunately, the online records available are rather limited; the Tox Foundation is listed as a California-registered corporation with Qureshi as its acting agent, but that does not preclude it being a non-profit entity. Complicating matters is the fact that, while Qureshi may be the Chief Financial Officer of the Tox Foundation company, irungentoo is supposedly the Chief Executive Officer.

This, too, is hard to verify online. Developer David Lohle left the project several months ago (following arguments about the management and transparency of finances). He posted an image of the forms allegedly filed to register the Tox Foundation, but admitted he had no way of knowing if the copy of the form he saw (sent to him by Qureshi) matched what was filed. In any case, blog posts and comments repeatedly refer to a Tox Foundation "board" that may not exist in any formal sense at all.

Concern over the status of the Tox Foundation has come up before—another former developer, Mark Winter, also cited disagreements about it in late 2014 as a reason he withdrew from the project. It is also clear that several contributors and community members have been upset that, whatever funds were collected (from various sources), money was not spent to improve the Tox code—such as by paying for a security audit.

For his part, Qureshi posted a message on the Tox.im blog on July 9. His account of the recent events claims that "the project" was unwilling to work with "The Tox Foundation" on infrastructure matters:

He concludes by noting that the tox.im wiki and mailing list will be put into read-only mode, that development and maintenance will continue for several Tox components, and that in two years he will reassess whether to continue or suspend the project.

It is always troubling to see a free-software project experience an acrimonious falling out, and is even more troubling when accusations of financial impropriety are at its core. Perhaps the only good news in this series of events is that irungentoo's relaunched Tox project seems to have the support of virtually the entire Tox community.

But, even there, the "entire Tox community" seems to have shrunk in recent months with the departures of Lohle, Winter, and a few other long-time contributors over the very lack of financial transparency that would appear to have made the current situation possible. The new project will likely have a uphill climb ahead of it to re-establish trust, and it may be in a precarious position with its 2015 class of GSoC students. Irungentoo claimed in his blog post to have learned a hard lesson about project governance and finances; wherever it is that Tox goes from here, its story will certainly be a lesson to many other software projects.

The Tox developers may have placed too much trust in a particular individual, but they also failed to set up a formal governance structure—complete with a well-known decision-making board (composed of multiple people) and, instead, relied on one person to file some sort of paperwork, the nature and specific contents of which remain mysterious even now. Furthermore, the Tox Foundation may have been set up to accept donations and payment, but there does not seem to be any formal agreement on the ownership of domain names and other pieces that make up the project infrastructure.

It is understandable that irungentoo and others felt out of their depth on these issues, but that is also why umbrella organizations like the Software Freedom Conservancy and Software in the Public Interest exist. They can help projects protect themselves against such conflicts, well before they reach the crisis stage.

Comments (7 posted)

Python 3.5 is on its way

It has been nearly a year and a half since the last major Python release, which was 3.4 in March 2014—that means it is about time for Python 3.5. We looked at some of the new features in 3.4 at the time of its first release candidate, so the announcement of the penultimate beta release for 3.5 seems like a good time to see what will be coming in the new release. Some of bigger features are new keywords to support coroutines, an operator for matrix multiplication, and support for Python type checking, but there are certainly more. Python 3.5 is currently scheduled for a final release in September.

As usual, the "What's New In Python 3.5" document is the place to start for information about the release. At this point, it is still in draft form, with significant updates planned over the next couple of months. But there is lots to digest in what's there already.

Type hints

One of the more discussed features for 3.5 only showed up in the first beta back in May: type hints (also known as PEP 484). It provides an way to optionally annotate the types of Python functions, arguments, and variables in a way that can be used by various tools. Type hints have been championed by Python benevolent dictator for life (BDFL) Guido van Rossum and came about rather late in the 3.5 development cycle, so there was always a bit of a question whether it would make the feature freeze that came with the first beta release.

But, just prior to that deadline, BDFL-Delegate Mark Shannon accepted the PEP on May 22, with a nod to some of the opposition to the feature that has arisen:

Shannon continued by noting that other languages have an operator-overloading feature that often get used in ugly ways, but that Python's is generally used sensibly. That's because of Python's culture, which promotes the idea that "readability matters". He concluded: "Python is your language, please use type-hints responsibly :)".

Matrix multiplication

A new binary operator will be added to Python to support matrix multiplication (specified by PEP 465). "@" is a new infix operator that shares its precedence with the standard "*" operator for regular multiplication. The "@=" operator will perform the matrix multiplication and assignment, as the other, similar operators do.

The @ operator will be implemented with two "dunder" (double underscore) methods on objects: __matmul__() and __rmatmul__() (the latter is for right-side matrix multiplication when the left-side object does not support @). Unsurprisingly, @= is handled with the __imatmul__() method. That is all the new feature defines, since Python does not have a matrix type, nor does the language define what the @ operator actually does. That means no standard library or builtin types are being changed to support matrix multiplication, though a number of different scientific and numeric Python projects have reached agreement on the intended use and semantics of the operator.

Currently, developers of libraries like NumPy have to make a decision about how to implement Python's multiplication operator (*) for matrices. All of the other binary operators (e.g. +, -, /) can only reasonably be defined element-wise (i.e. applying the operation to corresponding elements in each operand), but multiplication is special in that both the element-wise and the specific 2D matrix varieties are useful. So, NumPy and others have resorted to using a method for 2D matrix multiplication, which detracts from readability.

The only other use of @ in Python is at the start of a decorator, so there will be no confusion in parsing the two different ways of using it. Soon, users of NumPy will be able to multiply matrices in a straightforward way using statements like:

    z = a @ b
    a @= b

async and await

Another fast-moving feature adds new keywords for coroutine support to the language. PEP 492 went from being proposed in early April to being accepted and having its code merged in early May. The idea is to make it easier to work with coroutines in Python, as the new async and await keywords don't really add new functionality that couldn't be accomplished with existing language constructs. But they do provide readability improvements, which is one of the main drivers of the addition.

Essentially, functions, for loops, and with statements can be declared as asynchronous (i.e. may suspend their execution) by adding the async keyword:

    async def foo(...):
        ...

    async for data in cursor:
        ...

    async with lock:
        ...
	await foo()
	...

The await statement that can be seen in the with example is similar to the yield from statement that was added to Python 3.3 (described in PEP 380). It suspends execution until the awaitable foo() completes and returns its result.

While async and await will eventually become full-fledged keywords in Python, that won't happen until Python 3.7 in roughly three years. The idea is to allow programmers time to switch away from using those two statement names as variables, which is not allowed for Python keywords. As it turns out, await can only appear in async constructs, so the parser will keep track of its context and treat those strings as variables outside of those constructs. It is a clever trick that allows the language to "softly deprecate" the keywords.

Zip applications

Over the last few years there has been a lot of discussion on python-dev, python-ideas, and elsewhere about how to easily distribute Python programs; the zipapp module (specified in PEP 441) is a step along that path. Partly, the PEP authors simply want to promote a feature that was added back in the Python 2.6 days. It provides the ability to execute a ZIP format file that contains a collection of Python files, including a __main__.py, which is where the execution starts.

Besides just publicizing a "relatively unknown" language feature, the zipapp module would be added to provide some tools to help create and maintain "Python Zip Applications", which is the formal name for these application bundles. The bundles will have a .pyz extension for console applications, while windowed applications (i.e. won't require a console on Windows) will end in .pyzw . The Windows installer will associate those extensions with the Python launcher. It is hoped that .pyz will be connected with Python on Unix-like systems (by way of the MIME-info database), but that is not directly under the control of the Python developers.

The appropriate interpreter (i.e. Python 2 or Python 3) can be associated with the file by prepending a "shebang" line to the ZIP format (which ZIP archivers will simply skip). For example (from the PEP):

    #!/usr/bin/env python3
    #  Python application packed with zipapp module
    (binary contents of archive)

In addition, the zipapp module can be directly accessed from the command line. It provides two helper functions (create_archive() and get_interpreter()) but will likely be used as follows:

    $ python -m zipapp dirname

That creates a dirname.pyz file that contains the files in the directory (which must include a __main__.py). There are also command-line options that govern the archive name, the interpreter to use, or the main function to call from a __main__.py that zipapp will generate.

Formatting bytes

The distinction between bytes and strings in Python 3 is one of the defining characteristics of the language. That was done to remove some of the ambiguities and "magic" from handling Unicode that are present in Python 2. "Bytes" are simply immutable sequences of integers, each of which is in the range 0–255, while "bytearrays" are their mutable counterpart. There is no interpretation placed on the byte types; strings, on the other hand, always contain Unicode.

But there are a number of "wire protocols" that combine binary and textual data (typically ASCII), so when writing code to deal with those, it would be convenient to be able to use the Python string formatting operations to do so. Today, though, trying to interpolate an integer value into a bytes type does not work:

    >>> b'%d' % 5
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: unsupported operand type(s) for %: 'bytes' and 'int'

The adoption of PEP 461 will change that. All of the numeric formatting codes, when applied to bytes, will be treated as if they were:

    ('%x' % val).encode('ascii')

Where x is any of the dozen or so numeric formatting codes. In addition, %c will insert a single byte (either from an integer in the 0–255 range or from a byte type of length 1) and %b can be used to interpolate a series of bytes. Neither of those will accept string types since converting a string to bytes requires an encoding, which Python 3 will not try to guess.

In addition, bytes and bytearrays will be getting a hex() method. That will allow turning bytes into hexadecimal strings:

    >>> bytes([92, 83, 80, 255]).hex()
    '5c5350ff'

The grab bag

There are, of course, lots more features coming in Python 3.5—way more than we can cover here—but there are some additional ones that caught our eye and we will look at briefly. For example, the ordered dictionary in collections.OrderedDict has been reimplemented in C, which provides a 4–100x performance increase. PEP 485, which has been accepted for 3.5, proposes adding a function to test approximate equality to both the math and cmath modules: math.is_close() and cmath.is_close().

The "system call interrupted" error code on POSIX systems (EINTR, which gets turned into the Python InterruptedError exception) is often unexpected by applications when they are making various I/O calls, so those exceptions may not be handled correctly. It would make more sense to centralize the handling of EINTR. So, in the future, the low-level system call wrappers will automatically retry operations that fail with EINTR (which is a feature that is described in PEP 475).

The process of initializing extension modules (which are implemented as shared libraries), built-in modules, and imported modules is subtly different in ways that can lead to import loops and other problems. PEP 489 was proposed to regularize this process so that all of the different "import-like" mechanisms behave in the same way; it will be part of Python 3.5.

Eliminating .pyo files is the subject of PEP 488 and it will be implemented for the upcoming release. Those files were meant to hold optimized versions of the Python bytecode (unoptimized bytecode is usually placed into .pyc files), but the extension was used for multiple different optimization levels, which meant manually removing all .pyo files to ensure they were all at the same level. In the future, these __pycache__ file names will all end in .pyc, but will indicate both the optimization level and interpreter used in the name, for example: foo.cpython-35.opt-2.pyc.

Lastly, at least for our look, Python 3.5 will add an os.scandir() function to provide a "better and faster directory iterator". The existing os.walk() does unnecessary work that results in roughly 2x the number of system calls required. os.scandir() will return a generator that produces file names as needed, rather than as one big list, and os.walk() will be implemented using the new function. That will result in performance increases of 2–3x on POSIX systems and 8–9x for Windows.

As can be seen, there is lots of good stuff coming in Python 3.5. Over the next few months, testing of the betas and release candidates should hopefully shake out all but the most obscure bugs, leading to a solid 3.5 release in mid-September.

Comments (42 posted)