Guido on Python

Guido van Rossum's EuroPython 2015 keynote was billed as part prepared remarks, part Q&A, but he changed all that when he stepped up on the stage. Instead, it would all be Q&A, but he did prime the pump with some questions (and answers) of his own before taking audience questions. Topics included Python 3 (and 3.5), why there won't be a 2.8, why there are so many open bugs, PyPy, and what he hates about Python.

Django Girls

Van Rossum's first question to himself was about what he thought of Django Girls—the subject of the previous day's keynote. It was a great talk, he said, and he loved the storytelling. There will be none of that in his talk, nor any "pretty slides". He was stunned when he "heard that Ola ... or Ola ... had drawn the squirrels and badgers" for those slides.

Another aspect that he liked was their statement that they didn't know what they were doing. It reminded him of when he started working on Python 25 years ago. He didn't know what he was doing then, either. For example, he had no idea that a programming language needed a community with lots of different roles.

He was also quite impressed by the "strong brand" they have created in one year. "I predict that Ola and Ola, and Django Girls, will go really far."

Python versions

Shifting gears, his next query was on why developers would switch to Python 3. "Why can't you just give up on Python 3?", he asked himself rhetorically. But he is not saying that people should switch. He does want them to, but it is "a lot of hard work that could be spent on other things", such as features for their application or web site. Python 2.7 is not dead yet and will be supported with security fixes and, perhaps, security features for the next five years. Porting to Python 3 is a lot of grunge work, so why bother?

Python 3 is a "much better language" than Python 2, for one thing. It is much easier to teach. For example, the Django Girls workshops are all based on Python 3. That wouldn't have been possible if the Django developers hadn't done the grunge work to port the framework. That makes for a more pleasant first experience with the language (and the framework).

It is also getting better over time. There is "lots of cool new stuff" in Python 3.5, for example. Python 2 is a fine language and will remain exactly what it is, which is kind of asymptotically approaching 2.7-perfect, he said. The only way to benefit from all of the work that the core developers are doing is by moving to Python 3.

The perennial question of why not make a Python 2.8 release was next up, though Van Rossum noted that maybe that particular question was getting somewhat dated at this point. Python 2.8 wouldn't solve any of the problems that make people want it. Either it would have no new features, which means there would be no reason not to just stay at 2.7, or the floodgates get opened for backports from Python 3. That would make porting to 2.8 nearly what is needed to port to 3.

Unicode is, of course, the big hurdle to moving to Python 3. But, "enough is enough". So Python 2 is in a state where it will get no new features. That allows the core developers to focus their energy on making Python 3 better.

He then moved on to Python 3.5, which is due in September. He would have trouble choosing a favorite feature from the release because there are "way too many cool things" in it. For example, the performance improvement coming from os.scandir() is great, but it probably won't even be noticed by most. There is a small group of Python users that will be happy to see the new matrix multiplication operator. Numeric packages like NumPy and others will be able to start using it, which will allow writing matrix multiplication in a "much more natural fashion than calling a function".

Perhaps his favorite 3.5 feature should be type hints, since it is a PEP he worked on himself. It took a lot of work for the PEP to get accepted, which is a little bizarre since he is the benevolent dictator for life (BDFL) and could accept his own PEP. But he wanted to have independent review and acceptance of the PEP, which Mark Shannon was graciously willing to provide as the BDFL delegate, he said.

If you caught him unaware, though, he probably would name the new async and await keywords for coroutines as his favorite. It was the last PEP accepted for 3.5 and it provides a more natural way of specifying coroutines.

Open bugs

Someone recently asked him about all of the open bugs in the Python bug tracker. If you pick a random open bug, you will find that it probably has patches attached to it, a bunch of discussion, and even renowned core developers saying that the patches need to go in, but the bug is still not fixed. Is it because of an old boys' network or lame core developers? What needs to be done to get those patches applied?

The situation is the same for any large project, he said. Bugs that can't be closed right away, due to things like misread documentation, tend to pile up. They can be hard to reproduce due to the hardware or environment, for example. But those kinds of bugs don't have patches attached.

There are also bugs that are feature proposals that do have patches attached, but there is a general hesitation to accept changes like that because there is concern that they aren't useful, won't mesh with other similar language features, or that they will cause backward incompatibilities. It is hard to take patches without breaking things all the time.

In addition, the core developers all have a lot on their plates and no one is paid to merge patches for core Python. So if none of the core team cares about a particular patch or feature, they may not find the time to shepherd it through the merging process.

In a company, things are a little different. People are paid to do some amount of grunge work, but with open source you have to volunteer to do unpleasant tasks. Some core developers have been doing that for so long that they want a break from that kind of grunge work. These are some of the many reasons that there are lots of open bugs with long histories in the bug tracker.

Lastly, there is a statistical effect that many overlook. If you pick any bug at random, including closed bugs, you would likely get a closed bug. Many bugs are closed quickly and bugs that are easy to fix tend to get fixed quickly. But the average lifetime of an open bug grows linearly with the age of the project, he said.

The GIL

Someone from the audience asked about the global interpreter lock (GIL), looking for more insight into the problem and how it is being addressed. Van Rossum asked back with a grin: "How much time have you got?" He gave a brief history of how the GIL came about. Well after Python was born, computers started getting more cores. When threads are running on separate cores, there are race conditions when two or more try to update the same object, especially with respect to the reference counts that are used in Python for garbage collection.

One possible solution would be for each object to have its own lock that would protect its data from multiple access. It turns out, though, that even when there is no contention for the locks, doing all of the locking and unlocking is expensive. Some experiments showed a 2x performance decrease for single-threaded programs that didn't need the locking at all. That means there are only benefits when three or more threads and cores are being used.

So, the GIL was born (though that name came about long after it was added to the interpreter). It is a single lock that effectively locks all objects at once, so that all object accesses are serialized. The problem is that now, 10 or 15 years later, there are multicore processors everywhere and people would like to take advantage of them without having to do multiprocessing (i.e. separate communicating processes rather than threads).

If you were to design a new language today, he said, you would make it without mutable (changeable) objects, or with limited mutability. From the audience, though, came: "That would not be Python." Van Rossum agreed: "You took the words out of my mouth." There are various ongoing efforts to get around the GIL, including the PyPy software transactional memory (STM) work and PyParallel. Other developers are also "banging their head against that wall until it breaks". If anyone has ideas on how to remove the GIL but still keep the language as Python, he (and others) would love to hear about it.

PyPy

He was asked about PyPy, whether he used it and whether it might someday become the default interpreter. He does not use PyPy, but he does download it once in a while, plays with it for a few minutes, and likes what he sees. He uses Python in two modes, either writing a short little script to get something done, for which he just uses one of the interpreters he already has built on his system, or as a Dropbox engineer deploying Python code to its cluster.

The Dropbox cluster runs a modified Python 2.7, he said, which elicited audience laughter. "I said it, it is no secret", he said. Some parts of Dropbox do use PyPy where it is faster, but the company is worried that some subtle incompatibility will produce weird bugs that are hard to track down. "We have enough of those already".

PyPy shows that you can execute Python faster than CPython. It also provides a testbed where interesting ideas like STM can be tried. But conservative principles lead people to only use PyPy when they know they need the speed. The problem is that by the time you find that out, you have already deployed to enough machines that it makes switching hard. In that, it is like the problem with switching to Python 3.

Dropbox has a lot of third-party dependencies, some of which cannot even be rebuilt from the sources that it has. That is generally true of any company that has millions of lines of Python in production; he also saw it at Google. That also makes switching hard.

In summary, PyPy is a "really cool project", but there are checkboxes that it needs to satisfy that "are hard to check off". He half-jokingly suggested that perhaps PyPy needed to hire Ola and Ola from Django Girls to help create a bigger community around the project.

Favorites

Five short questions about his favorites was up next. Favorite web framework? He said he only writes one web app in any framework and that the last he tried was Flask. Favorite testing library? He mostly just uses unittest and mock from the standard library. Editor? He uses Emacs, but started out with vi (both of which were greeted with applause, presumably by different sets of audience members). He still uses vi (or Vim) occasionally, but if he does that for five minutes, it takes him fifteen minutes to get used to Emacs again.

What is his favorite language besides Python? He used to say C, but "that's kind of boring". People he trusts tell him that modern C++ is a good language. He likes Go, but hasn't written anything significant in it. From talking with the designers, he likes the looks of Swift, which stole a bunch of things from Python. It is easy to steal bad stuff from languages you like and end up with an incoherent pile of features, but Swift's designers appear not to have done that. Finally, favorite exception? He chuckled and answered KeyboardInterrupt to more applause and laughter.

And what he hates

The final question was about what he hates in Python. "Anything to do with package distribution", he answered immediately. There are problems with version skew and dependencies that just make for an "endless mess". He dreads it when a colleague comes to him with a "simple Python question". Half the time it is some kind of import path problem and there is no easy solution to offer.

With that, his time was up. The EuroPython organizers had a gift for each of the keynote speakers: a Basque beret and bandana. They were presented to Van Rossum at the end of the talk (seen above at right).

Comments (13 posted)

A look at the Fotoxx image editor

Note: several of the project sites linked to below are temporarily offline, evidently due to a storage fault incident at SourceForge. Hopefully they will be back online shortly.

Image editing, like a great many tasks, tends to be dominated at any one time by a small handful of applications. For example, at the moment, the most popular programs for editing photos are probably GIMP, darktable, digiKam, and RawTherapee. A few years ago, the list might have included Rawstudio, F-Spot, Geeqie, and gThumb. Tastes change; individual projects gain and lose steam, and what counts as state-of-the-art technology shifts over time. Nevertheless, at any given time, most of the current crop of applications tend to adhere to the same general approach where user-interface and interaction decisions are concerned. So it is all the more intriguing to encounter a project like Fotoxx, an image editor that takes its own approach. The latest release recently became available, and offers an interesting editing experience.

Fotoxx has been in development since mid-2008. The latest release, version 15.07, was unveiled in early July. Packages are available in Debian format and there are the requisite source-code bundles for other distributions. The only external dependency that might cause some trouble is the raw-photo processing package UFRaw. UFRaw is most often used as a plug-in for GIMP, so it might need to be installed for users who favor some other raw processor.

In some respects, Fotoxx is a standard image-editing tool. It lets the user index their image collection (in place, so images are not needlessly duplicated), browse and search through it, and perform a wealth of corrections and transformations on individual images or in batches.

What makes Fotoxx distinct, though, is its atypical user interface, which has seemingly evolved over the years along a path completely different from that of its contemporaries. Fotoxx uses GTK+ as its GUI toolkit, but it does not sport a tool palette or control panels that resemble any of those used by GIMP, darktable, and the like. In fact, it eschews any number of standard GTK+ practices—such as having a menu bar at the top of the window.

Instead, there is a single strip of buttons down the left-hand side of the application window. Clicking on a button pops up a floating text menu of available features, and almost all of the features open their own pop-up windows filled primarily with text-labeled buttons rather than graphics. For some tools, this is not a big change. Adjusting the hue of the image, for example, requires only a few sliders: the effects of moving those sliders are visible immediately on the image, so no other UI elements are required.

For other tools, the text-label approach is more challenging. The paint/clone tool, for instance, uses text labels to indicate the radius and transparency level of the paintbrush. GIMP's approach is clearer, simply showing a circle of the appropriate size and opacity. The text-driven UI is also interesting because it has some absolute limits: there are some controls where only a graph will do, such as adjusting brightness or tone curves. For those tools, Fotoxx displays a graph.

The other manner in which the Fotoxx UI differs from its contemporaries is that almost all of the pop-up windows it uses to provide features are modal. That is, they sit on top of the main image window and take control of the interface; until you are done with one tool or filter and close its window, you cannot open a different tool. This design decision leads to some interesting side-effects—such as a tendency toward a step-by-step, linear processes to perform editing tasks, as opposed to image tools that you can open up and fiddle with haphazardly.

Some of the standard editing features certainly suffer as a result. When selecting part of an image, Fotoxx pops up a window filled with checkboxes and text buttons that correspond to the various selection modes (drawing boxes, drawing freehand lines, select-by-color, and so on). You can construct your selection area by alternating between the tools available, drawing individual selection regions that appear on the image as red outlines. When you are done, you click on the "Finish" button, then must individually click in each zone of the image to activate the selection. When selecting complex or compound shapes, this becomes tedious quickly.

On the other hand, some features benefit from the approach. For example, the perspective-correction tool is among the simplest of any image editor. You merely click on four points in the image (in any order), then click "Apply" to transform the image so that the quadrilateral you have chosen becomes a proper rectangle. The corresponding GIMP tool uses a floating grid rendered on top of a preview image—which can be difficult to see through and unresponsive to manipulate.

Apart from the novel user interface (which can certainly be deemed a plus or a minus, depending on the individual user's inclinations), Fotoxx excels in the areas of speed and simplicity. Image operations are fast and, importantly, that speed includes showing an immediate preview of the effect in question on the open image. Many of GIMP's effects, by comparison, provide only a thumbnail-sized preview of their output. That can inadvertently add time as you adjust the settings and sliders, then must pan and zoom around the thumbnail preview to see if your changes produce the desired outcome.

Naturally, simplicity is a double-edged sword. Darktable and GIMP (especially GIMP combined with the G'MIC plugin) offer a dizzying array of filters and effects. For simple editing of photos that do not suffer from major defects, darktable and GIMP can be on the sluggish side. Applications that focus on "image-collection management" (such as digiKam and GNOME Photos) generally offer quick-adjustment features, but those applications also have a tendency to try taking over one's image collection even when not asked—scanning your home directory and firing up databases and indexing processes.

Fotoxx fills in nicely for these situations. It does not try to force you into indexing your complete photo archive (or, worse yet, copying it to the application's preferred location), but it provides access to color and tone adjustments, sharpness and noise filters, and standard retouching tools. And, if the adjustments needed do go beyond what Fotoxx itself can handle, the application allows you to open the image in GIMP by clicking on the "Tools" -> "Plugins" -> "GIMP" option.

This is not to say that Fotoxx is lacking in powerful features. Recent releases have added a "World Maps" feature that lets you view all of your geotagged images on a global, zoomable map. There is also support for some specialty image-editing operations like exposure stacking (which is used, for example, to combine multiple dim astrophotography images into a single photo) and panorama stitching. Such specialized features, though, tend to require the user to make a large time investment both behind the camera and in front of the keyboard, so comparing them to the competition (e.g., Hugin) objectively can be difficult.

For everyday editing, however, Fotoxx is well worth keeping around. Its peculiar interface may occasionally cause some frustration, but it also prompts some "why haven't other projects thought of that?" moments.

Comments (6 posted)

Django Girls one year later

Though it got a bit of a late start due to some registration woes, the first day of EuroPython 2015 began with an engaging and well-received keynote. It recounted the history of a project that got its start just a year ago when the first Django Girls workshop was held at EuroPython 2014 in Berlin. The two women who started the project, Ola Sitarska and Ola Sendecka, spoke about how the workshop to teach women about Python and the Django web framework all came together—and the amazing progress that has been made by the organization in its first year.

It all started when Sitarska and Sendecka met while organizing DjangoCon 2013 that was held in their home country of Poland. They decided to hold the event in a circus tent at a horse racing track in Warsaw. They were able to make that event work, which provided a strong bond that obviously continues to this day. Sendecka said that experience is a great way to get to know someone and that they both recognized they could pull off "crazy things" together.

It is, they noted, quite a coincidence that their names are the same length and only differ by a few letters; both also come from the same region of Poland (Silesia). They now live in London and both work as Django developers for a company called Potato. Perhaps unsurprisingly, some people think there is only one "Ola", which results in a fair amount of misdirected email and other types of confusion.

A fable

As a way of introducing Django Girls, they began with a fable of sorts that was illustrated with some eye-catching drawings made by Sendecka (slides at Speaker Deck). The story was about "Liz", who is a squirrel who is fascinated by technology and programming in particular. As she makes her way into the "forest of technology", though, she finds that there are few squirrels who are interested in technology. In fact, most of those who are learning about technology in the forest were all very similar to each other, but much different from her.

In fact, the others were badgers, she discovered. She wanted to fit in, but always stood out from the badger pack. They were generally nice, and she had nothing against badgers, but she could feel the eyes of the badgers on her whenever she entered a room and she was often "complimented" with lines like: "You are not like other squirrels" or "You are pretty good at technology for a squirrel."

She realized that the badgers knew lots about technology, so she thought she should perhaps become more badger-like to fit in. But that didn't change anything, she was still an outsider. At one point she organized a technology event and was asked by a badger whether she was just there for social reasons. She constantly had to prove her knowledge of technology, while badgers who knew far less than her were able to speak up freely.

Liz started to lose confidence in herself, worried that she was too shy and reserved to ever fit into the technology crowd, even though she loved the technical aspects of the field. She was able to make some friends among the badgers, though when she looked closely, she realized they weren't really badgers. Instead, they were really various other types of animals that only just kind of looked like badgers. The support from these friends helped her push on with her studies.

After a while, Liz realized that the problem wasn't with her, it was with the technology forest itself. She had to do so much more than badgers did to get the same recognition, but when she brought that subject up, it was not well-received, or even believed, by the badgers. She asked around about the problem on the internet and got the predictable responses about squirrel's inherent inferiority and inability to ever truly master technology.

Liz despaired of the situation ever changing. She was too tired to fight every day for her place, but then, through a fortunate series of events, she met another squirrel who loved technology. This squirrel was also named "Liz" and they became fast friends. She finally had someone with whom she could share her frustrations of trying to live in the forest of technology. She realized that she is not the only squirrel who loved technology, but was as intimidated and scared by the badger's world as she was.

Liz and Liz started to think about how awesome it would be if there were more squirrels that could get involved with technology and who could learn to love programming the way they did. So they came up with a crazy plan to teach other squirrels how to code. They told all of their friends about the idea and with their support, and the help of lots of others who joined in, the idea became a reality.

In the year since then, thousands of squirrels have learned how to program. It is all because two little squirrels met at the right time and became excited about this crazy idea. Today, they said, Liz is standing on the stage at EuroPython telling the story of this journey and of a dream that became a reality. That dream is Django Girls.

Teaching Python and Django

Sitarska and Sendecka started organizing the first Django Girls workshop just two months before last year's conference. It was targeted at teaching women who were complete beginners and had "never written a line of code in their life". In the year since, they have seen over 1600 women attend these workshops from "literally all over the world".

After a meeting with one of the EuroPython organizers, Sitarska got the idea to run a workshop. She immediately sent Sendecka email asking if she wanted to help. In something of a massive understatement, in the email she said "I don't think it's gonna be a lot of work ...". Sendecka responded "she lied" with a laugh. The idea was to get twenty students and six coaches for the workshop and to use an existing Django tutorial from the internet.

So they started working on putting the workshop together that day. But, most of the time, "we had absolutely no idea what we were doing". The amount of work started to snowball. They came up with the idea of providing financial aid for attendees, which required more sponsors and figuring out how to transfer money to the students.

They also realized that they didn't like any of the tutorials that were out there. All of them assumed that the reader was already a programmer and knew what a web framework is, what a server is, and so on. They ended up writing their own tutorial, which turned out to be roughly 90 pages of text.

In addition, they promoted the workshop in all of the places they could think of and ended up spending a lot of time evaluating the 300 attendee applications. Eventually the workshop expanded to 45 attendees, which meant that 15 coaches were needed as Django Girls has a 3:1 ratio for students to coaches.

The Django Girls tutorial has now been read by over 94,000 people at this point, which is 1/4 the population of Bilbao (where EuroPython 2015 was held), they noted. In total, 1,646 women have learned Python and Django in these workshops. More than 70 have now been held in 34 countries on 6 continents. Nothing, so far, in Antarctica, but "we are working on that too", Sendecka said with a chuckle.

News

The project has grown so much that a legal entity was needed to administer it, so the Django Girls Foundation was started in June. Django Girls has been so successful because of the huge numbers of volunteers who have put in their own time as organizers (137), contributors to the tutorial (111), and as part of a "huge army of coaches" (548). It came to a point where overseeing all of that became too much, so they asked for help and got four more people who are now helping to administer the project; they "help keep us sane".

That still is not enough, they realized, so they have made the big decision to hire the first paid position for the foundation. They are looking for an "Ambassador of Awesomeness" to help ensure that the organization does the best work that it can. It was a bit of a terrifying decision, especially since "we still don't know what we are doing and are making it up as we go". But they are sure it is the right decision to make to ensure that Django Girls "is sustainable and can grow beyond us".

They believe that things are just getting started for the project and that it can have an even bigger impact down the road. They have another big project in the works that they shared for the first time at the conference. The tutorial was enjoyable to write and "to our surprise we must have done a pretty good job". They get many requests for more advanced tutorials or ones that cover other parts of computer science. The tutorial was necessarily geared toward how much can be taught in a single day, so it skimmed over lots of material to make it all fit.

But there is much more about computers and the internet that can be taught, so they are in the process of writing a book. It will try to capture the style of the workshops but cover more material in a book that will be "beautifully crafted". The book is called Yay Python and those interested can follow the progress at that web site. The announcement was met with a loud round of applause and a stage whisper that there is "no going back on that now".

The book will build on the Django Girls tradition, with "a good dose of emoji and funny little quirks". It is still a tutorial on how to build a Django web site, but there will be lots more material, including chapters on topics like "how a computer works" and a "connect the dots" exercise to explain HTTP requests and responses. It will also talk about the communities behind Python and Django, along with how open source works. The idea is to help people "fall in love with the internet as we did".

One rule

The one rule they have adhered to throughout the Django Girls process is to "always go the extra mile" and the details of the workshops show this, they said. The room is decorated with flowers and balloons, rather than being a "boring classroom" and they do fun things like cupcake tasting or yoga during the day. To ensure that organizers have all of the information they need to run the event, and that attendees have a similar experience no matter where the event is held, there is an organizer's guide. There is also a coaching manual. All of that material is open source, so that it can be repurposed as needed. It "took weeks to write down" all of that information, but they want to ensure that Django Girls is bigger than just the two of them.

Workshops are meant to be fun and friendly. Enthusiasm from the organizers, coaches, and participants helps with that. "The power of enthusiasm is huge", they said. When the two of them get excited, that gets passed on to organizers and from there to participants. That is part of what has made Django Girls so successful so quickly.

But every labor of love comes "with a huge emotional cost". It is an enormous job and one that they always feel they are not quite putting enough of themselves into. The thing that keeps them going when they are feeling that way, though, is the stories from women who have literally changed their lives by attending the workshops.

For example, there is Dori, who was sitting in the keynote audience somewhere, and was attending her second EuroPython. At the pilot workshop, she spent half of the time in the workshop trying to get the Django server to play nicely with her Hungarian keyboard. At the time, they thought she had "the worst experience ever" and would probably never want to have anything to do with Django or programming ever again. But today she is a Django developer in Budapest, has organized a monthly meeting for Python developers, and organized two Django Girls workshops there.

Another was Josie, who was only 13 when she attended the first workshop with her mother. Immediately after that, she did a lightning talk at EuroPython in front of a thousand people about her Django Girls experience and her plan to organize the Pyxie Dust project to create a similar workshop for girls her age. That event took place in Zagreb, Croatia in 2014.

Or Linda, who organized a workshop in Nairobi, Kenya just a few months after the initial workshop in Berlin. She "blazed a trail" for other women who organized workshops in Africa after that, they said. They talked about several other women who have "graduated" from attending the workshop to organizing and coaching at others, as well as getting involved in the Python and Django communities in other ways. The Django Girls blog has lots of similar stories as well as event reports and other news.

They concluded their talk with a "huge thank you" to everyone that has helped out along the way. Their dream is for Django Girls not to be needed at some point. It may take 5, 10, or 15 years, but if that can happen, they will have achieved everything they set out to do. That is the ultimate goal for the project.

One thing they learned over the last year is that "kindness is contagious". If you put a lot of energy and effort into something to help others, "the universe will find a way to pay it all back to you". They encouraged others to follow in their footsteps; to find their dream and make it happen. All that it took was "two little squirrels with a big, big dream and a generous, wonderful community" to create something like Django Girls. Remember that "you are not alone, you are among friends", they finished—to thunderous applause.

A YouTube video of the talk is also available.

Comments (10 posted)

Assessing risk with the Core Infrastructure Initiative

The Linux Foundation's Core Infrastructure Initiative (CII) exists to "fortify" critical open-source software projects with funding, code reviews, and other kinds of support, with a particular eye toward shoring up those packages to prevent serious security crises. CII was formed in response to the memorable "Heartbleed" vulnerability discovered in OpenSSL, which was the first adopted project. Recently, CII unveiled its Census Project, a semi-automated ranking of open-source projects by security risk. The numbers make for some interesting reading—although the conclusions subsequently drawn by the CII can be puzzling.

The Census Project was announced on July 9, at which time the CII presented the results of its project-analysis work. There is a multi-page HTML table on the Census Project page, as well as a white paper [PDF] co-authored with the Institute for Defense Analyses (IDA) that goes into detail on the methods and metrics considered and used. The upshot is that each project examined in the census is assigned an integer score on a scale from 0 to 16, with higher numbers indicating the greatest risk that the project could be the source of an undiscovered security hole. The peculiar aspect to the story, however, is that the CII appears to have amassed a list of high-risk projects that has little to do with the results of the Census Project.

Scoring projects

The process used to determine the scores, though, did not involve any inspection of the code itself—just a look at project "metadata" of various flavors. As described on the web site, the Census Project counts seven factors when compiling its scores. How these factors are measured requires a more detailed examination (below), but the list itself is short:

• The number of CVEs filed (worth from 0 to 3 points)
• The project's contributor count over the past 12 months (2 to 5 points)
• The project's ranking in the Debian popularity list (point value unspecified)
• Whether or not the project has a known web site (0 or 1 point)
• Whether or not the package is exposed to the network (0 or 2 points)
• Whether or not the package processes network data (0 or 1 point)
• Whether or not the package could be used for local privilege escalation (0 or 1 point)
• Whether or not the project includes an executable or only provides data (0 or -3 points)

The number of points assigned for popularity in Debian is not specified. The other factors, however, are only enough to add up to a score of 13, so perhaps the popularity is a 0–to–3 score—and it would appear that a high popularity ranking corresponds to more "risk" points. In addition, CII's Emily Ratliff noted that only CVEs since 2010 were counted.

Individual pages for each project assessment provide a bit more detail (see, for example, the page for tcpd), noting which language the program is implemented in, so other factors may be part of the scoring formula. Ultimately, of course, the score is the product of a human assessment of the project, as the CII web site makes plain. While some of the input data is harvested from Debian and from Black Duck's OpenHub, other factors clearly involve some qualitative judgment—such as whether or not a package could be used for local privilege escalation—and the white paper mentions that the speed with which CVEs are fixed played a role in the rankings.

Of the packages assessed so far, the first big cliff in the scoring occurs between the packages scoring 9 or above and those scoring 8 or below. This top-scoring class of packages includes the following:

Package	Score
tcpd	11
whois	11
ftp	11
netcat-traditional	11
at	10
libwrap0	10
traceroute	10
xauth	10
bzip2	9
hostname	9
libacl1	9
libaudit0	9
libbz2-1.0	9
libept1.4.12	9
libreadline6	9
libtasn1-3	9
linux-base	9
telnet	9

Regrettably, the raw numbers that make up each package's score do not appear to be available. It would have been interesting to see the exact point values assigned for number of contributors, for example. It is also not entirely clear how some of factors are scored—does "could be used for local privilege escalation" mean simply "is installed setuid," for example? The project has a GitHub repository where some of the data-scraping code can be inspected, but the CII site and white paper both indicate that human assessment of the data plays a major role in the final process (starting with cleaning up the "noisy" raw data).

Beyond scores

In the end, though, the oddest thing about the scoring is that these raw scores do not indicate which projects CII will invest in. The white paper, after a lengthy (60-page) explanation of the methodologies employed, comes up with a different set of human-selected "riskiest" projects based on the authors' "knowledge of how the programs are used" and on which project "appear to be relatively unmaintained". The human-identified project list includes: xauth, bzip2, libaudit0, libtasn1-3, bind9, exim4, isc-dhcp, gnutls26, gpgme, openldap, pam, openssl, net-tools, openssh, rsyslog, wget, apr-util, coolkey, ntp, gnupg, gzip, expat, freetype, libgcrypt11, keyutils, xz-utils, p11-kit, pcre3, cyrus-sasl2, libxml2, shadow, tar, zlib, apr, libjpeg8, libpng, libressl, unzip, giflib, mod-gnutls, postfix, and cryptsetup.

This list contains little that is surprising. The projects highlighted are those that must deal with untrusted network connections, those that are responsible for processing potentially malicious data file formats, and those that are responsible for enforcing security measures for the system as a whole or for application programs. This may seem a bit anticlimactic, since it varies little from the list that any security-conscious user might come up with on their own.

Nevertheless, it is good to see someone attempt systematic analysis to reach a conclusion about the riskiness of common programs. The troubling factor is that, so far, the analysis only underscores common sense. The larger question is what CII intends to do with this information. The first few CII-supported projects (ntpd, GnuPG, Frama-C, OpenSSL, OpenSSH, Debian's reproducible builds, and The Fuzzing Project) were selected before there was a formal process in place.

The Census Project is a first step toward assembling such a process. Still, the web page makes a point of saying that "the decision to fund a project in need is not automated by any means." The white paper concludes by saying only that CII participants "believe the next step is to further investigate these OSS projects for security and project healthiness."

Interestingly enough, outsiders are invited to participate in the CII's project-identification process by contributing patches or suggestions to the Census Project code on GitHub or by writing to one of the CII mailing lists. Thus far, two other projects have been suggested for consideration on the cii-census list (the archives of which are visible only to subscribers): the Trinity fuzz tester and the PaX patch set. Both suggestions were referred to the CII steering committee, which includes one representative each from the supporting companies: Amazon Web Services, Adobe, Bloomberg, Cisco, Dell, Facebook, Fujitsu, Google, Hitachi, HP, Huawei, IBM, Intel, Microsoft, NetApp, NEC, Qualcomm, RackSpace, Salesforce.com, and VMware.

The CII itself is still finding its footing. Apart from the Census Project, multiple pages on the site invite projects interested in funding to contact the CII with a grant request, but point them to a contact page but does not yet have a formal process defined. Time will tell how CII goes about selecting which projects to support from among the high-risk prospects. Hopefully, much of that selection process will take place in the open. As this census shows, there is no shortage of important projects that are in need of additional support; transparency in determining which ones merit support is as important as the ability to study the resulting improvements to the source code.

Comments (4 posted)

Security quotes of the week

Comments (none posted)

apache: multiple vulnerabilities

Comments (none posted)

cacti: SQL injection

Comments (none posted)

drupal7-feeds: three vulnerabilities

Comments (none posted)

drupal7-migrate: cross-site scripting

Comments (none posted)

drupal7-views_bulk_operations: permission bypass

Comments (none posted)

freexl: denial of service

Comments (none posted)

groovy: code execution

Comments (none posted)

inspircd: denial of service

Comments (none posted)

java: multiple unspecified vulnerabilities

Comments (none posted)

java: multiple unspecified vulnerabilities

Comments (none posted)

libidn: information disclosure

Comments (none posted)

lxc: two vulnerabilities

Comments (none posted)

mysql: multiple unspecified vulnerabilities

Comments (none posted)

mysql: multiple unspecified vulnerabilities

Comments (none posted)

pacemaker: privilege escalation

Comments (none posted)

pdns: denial of service

Comments (none posted)

php-horde: multiple vulnerabilities

Comments (none posted)

pki-core: cross-site scripting

Comments (none posted)

python-keystonemiddleware: certificate verification botch

Comments (none posted)

rawstudio: insecure temp files

Comments (none posted)

rsyslog: denial of service

Comments (none posted)

ruby: denial of service

Comments (none posted)

springframework: denial of service

Comments (none posted)

squashfs-tools: two vulnerabilities

Comments (none posted)

tidy: two vulnerabilities

Comments (none posted)

webkitgtk4: denial of service

Comments (none posted)

xen: privilege escalation

Comments (none posted)

Kernel release status

Stable updates: 4.1.3 and 4.0.9 were released on July 21.

Comments (none posted)

Quotes of the week

/*
 * Well, when the cobbler got mad like this, he would go into hiding.  He
 * would not make or sell any boots.  He would not go out at all.  Pretty
 * soon, the coffee shop would have to close because the cobbler wasn't
 * coming by twice a day any more.  Then the grocery store would have to
 * close because he wouldn't eat much.  After a while, everyone would panic
 * and have to move from the village and go live with all their relatives
 * (usually the ones they didn't like very much).
 *
 * Eventually, the cobbler would work his way out of his bad mood, and
 * open up his boot business again.  Then, everyone else could move back
 * to the village and restart their lives, too.
 *
 * Fortunately, we have been able to collect up all the cobbler's careful
 * notes (and we wrote them down below).  We'll have to keep checking these
 * notes over time, too, just as the cobbler does.  But, in the meantime,
 * we can avoid the panic and the reboot since we can make sure that each
 * subtable is doing okay.  And that's what bad_madt_entry() does.
 */

Comments (none posted)

Gorman: Continual testing of mainline kernels

Comments (6 posted)

rm -r fs/ext3

A few eyebrows went up when Jan Kara posted a patch removing the ext3 filesystem recently. Some users clearly thought the move represented a forced upgrade to ext4; Randy Dunlap remarked that "this looks like an April 1 joke to me". In truth, it is neither a joke nor a forced upgrade; it is, however, an interesting story to look back at.

Nine years ago, in the middle of 2006, the premier filesystem for most users was ext3, but that filesystem was showing its age in a few ways. Its 32-bit block pointers limited maximum filesystem size to 8TB, a limit that was not too restrictive for most users at the time, but which would be highly problematic today. The filesystem tracks blocks in files with individual pointers, leading to large amounts of metadata overhead and poor performance on larger files. These problems, along with a number of missing features, had long since convinced developers that something newer and better was required.

For a while, some thought that might be a filesystem called reiser4, but that story failed to work out well even before that filesystem's primary developer left the development community.

The ext3 developers came up with a number of patches aimed at easing its scalability problems. These patches were made directly against the ext3 filesystem, with the idea that ext3 would evolve in the direction that was needed. There was, however, some resistance to the idea of making major changes to ext3 from developers who valued that filesystem in its current, stable form. One of those developers, it turned out, was Linus who, as we all know, has a relatively strong voice in such decisions.

And so it came to be that the ext3 developers announced their intent to create a new filesystem called "ext4"; all new-feature development would be done there. Actually, the new filesystem was first called "ext4dev" to emphasize its experimental nature; the plan was to rename it to "ext4" once things were stable, "probably in 6-9 months". In the real world, that renaming happened nearly 28 months later and was merged for the 2.6.28 kernel.

Since then, of course, ext4 has become the primary Linux filesystem for many users. It has seen many new features added, and it is not clear that this process will stop, even though ext4 is now in the same position that ext3 was nine years ago. Through this entire history, though, ext4 has retained the ability to mount and manage ext2 and ext3 filesystems; it can be configured to do so transparently in the absence of the older ext2 and ext3 modules. And, indeed, many distributions now don't bother to build the older filesystem modules, relying on ext4 to manage all three versions of the filesystem.

Back when ext4 was created, it was envisioned that the older filesystem code would eventually become unnecessary. The plan was that when this happened, "perhaps 12-18 months out", the ext3 code would be removed. Once again, reality had something different to say, and the ext3 code endured for over nine years. Unless something surprising happens, though, that record is about to come to an end; ext3 could be removed as soon as the 4.3 development cycle, taking some 28,000 lines of code with it. And most users, even those with ext3 filesystems, will not even notice.

One might well wonder whether we will see a similar story in the future and the addition of an ext5 filesystem. For the time being, that does not seem to be in the works. Ext4 has picked up a number of features in recent years, with encryption as the most recent example, but there has been no talk of moving development to a new source base. Over the years, perhaps, the ext4 developers have done well enough at not breaking things that users are less worried about new development than they once might have been.

At the other end, there is the question of the ext2 filesystem. That code, too, could be replaced by ext4, but there seems to be no pressure to do so. Ext2 is small, weighing in at less than 10,000 lines of code; ext3 and the associated JBD journaling code come in at 28,000, while ext4 and JBD2 add up to closer to 60,000 lines. The simplicity of ext2 makes it a good filesystem for developers to experiment with, and its maintenance cost is nearly zero. So there is no real reason to take it out anytime soon.

Ext3, being rather larger than ext2, is a more promising target to remove, though Jan said that its maintenance cost was pretty low. The fact that this code has been so thoroughly replaced makes the removal decision relatively easy — but that decision still took nine years to come about. Even so, if all old kernel code were this easy to get rid of, the kernel would be quite a bit smaller than it is today.

Comments (18 posted)

Atomic additions

Atomic logical operations

The simpler addition is the atomic logical operations patch set from Peter Zijlstra. Peter noted that there was no notion of logical operations on atomic_t variables that was the same across all architectures. Some of them have related operations called atomic_set_mask() and atomic_clear_mask(), but those operations are defined inconsistently across architectures when they are present at all.

To clean this situation up a bit, Peter introduced these new operations:

    void atomic_and(int mask, atomic_t *value);
    void atomic_or(int mask, atomic_t *value);
    void atomic_xor(int mask, atomic_t *value);
    void atomic64_and(int mask, atomic64_t *value);
    void atomic64_or(int mask, atomic64_t *value);
    void atomic64_xor(int mask, atomic64_t *value);

There is also a pair of simple wrappers (atomic_andnot() and atomic64_andnot()) that simply flip the bits of the mask argument. All of these functions have a void type; there are no _return variants (e.g. atomic_and_return()) that return the result of the operation at the same time. Uses of atomic_set_mask() and atomic_clear_mask() in the tree are changed to use the new functions, and the old ones have been deprecated.

Relaxed atomics

Atomic operations do not normally function as memory barriers; in other words, the processor and the compiler are both free to reorder atomic operations relative to other operations in ways that could create confusion in concurrent situations. The exception to that rule is the _return operations; for example, atomic_add_return() will add a value to an atomic_t, return the resulting value, and function as a full memory barrier.

Those rules are looking increasingly inadequate when faced with the growing complexity and concurrency of contemporary systems. All-or-nothing memory barriers are an overly blunt tool for developers who are working to maximize concurrency and minimize the cost of the associated operations. What developers would like to see instead is the ability to explicitly control barriers with "acquire" and "release" semantics.

For those who don't want to do a quick read through the increasingly scary memory-barriers.txt file, here is a quick refresher. An "acquire" operation (usually a read) contains a barrier guaranteeing that the operation will complete before any subsequent reads or writes. A "release" operation (normally a write) guarantees that any reads or writes issued prior to the release will complete before the release operation itself completes. Acquire and release operations are thus only partial barriers. In many situations, though, they are all that is needed, and they can be less expensive than full barriers; developers seeking to maximize performance thus want to use them whenever possible.

Will Deacon set out to provide that control with atomic operations. The result was a new set of atomic operations:

    int atomic_read_acquire(atomic_t *value);
    void atomic_set_release(atomic_t *value);

    int atomic_add_return_relaxed(int i, atomic_t *value);
    int atomic_add_return_acquire(int i, atomic_t *value);
    int atomic_add_return_release(int i, atomic_t *value);

    int atomic_sub_return_relaxed(int i, atomic_t *value);
    int atomic_sub_return_acquire(int i, atomic_t *value);
    int atomic_sub_return_release(int i, atomic_t *value);

    /*
     * And so on for atomic_xchg(), atomic_cmpxchg(),
     * xchg(), and cmpxchg().
     */

Will's patch also defines the 64-bit and atomic_long_t versions of the above functions. In each case, the "bare" version of the name (e.g. atomic_add_return() gives full-barrier semantics, while the _relaxed version provides no barrier at all. In between are the versions that include barriers with acquire or release semantics.

The first use of these new primitives is with the queued reader/writer lock code. Assuming they are merged, they will likely find their way into other performance-sensitive parts of the kernel in short order. That should be good for the speed of the system (though no benchmark numbers have been posted), but it comes at the cost of requiring more developers to understand the details of how the barrier semantics work. It is becoming increasingly hard to hide these details in architecture-specific code over time. As the complexity of our systems grows, the complexity of the software will have to increase as well.

Comments (11 posted)

Domesticating applications, OpenBSD style

It is fair to characterize the sandboxing features in Linux as being relatively complex. The complexity of the security module options, and SELinux in particular, is legendary. The seccomp() system call has two modes: very simple (in which case almost nothing but read() and write() is allowed), or rather complex (a program written in the Berkeley packet filter (BPF) language makes decisions on system call availability). There is a great deal of flexibility available with both security modules and seccomp(), but it comes at a cost.

OpenBSD leader Theo de Raadt is particularly scornful of the BPF-based approach:

His posting contains a work-in-progress implementation of a simpler approach to sandboxing (mostly written by Nicholas Marriott, it seems) in the form of a system call named tame().

The core idea behind tame() is that most applications run in two phases: initialization and steady-state execution. The initialization phase typically involves opening files, establishing network connections, and more; after initialization is complete, the program may not need to do any of those things. So there is often an opportunity to reduce an application's privilege level as it moves out of the initialization phase. tame() performs that privilege reduction; it is thus meant to be placed within an application, rather than (as with SELinux) imposed on it from the outside.

The system call itself is simple enough:

    int tame(int flags);

If flags is passed as zero, the only system call available to the process thereafter will be _exit(). This mode is thus suitable for a process cranking on data stored in shared memory, but not much else. For most real-world applications, the reduction in privilege will need to be a bit less heavy-handed. That is what the flags are for. If any flags at all are present, a base set of system calls, with read-only functionality like getpid(), is available. For additional privilege, specific flags must be used:

• TAME_MALLOC provides access to memory-management calls like mmap(), mprotect(), and more.

• TAME_RW allows I/O on existing file descriptors, enabling calls like read(), write(), poll(), fcntl(), sendmsg() and, interestingly, pipe().

• TAME_RPATH enables system calls that perform pathname lookup without changing the filesystem: chdir(), openat() (read-only), fstat(), etc.

• TAME_WPATH allows changes to the filesystem: chmod(), openat() for writing, chown(), etc. Note that TAME_RPATH and TAME_WPATH both implicitly set TAME_RW as well.

• TAME_CPATH allows the creation and removal of files and directories via rename(), rmdir(), link(), unlink(), mkdir(), etc.

• TAME_TMPPATH enables a number of filesystem-related system calls, but only when applied to files underneath /tmp.

• TAME_INET allows socket() and related calls needed to function as an Internet client or server.

• TAME_UNIX allows networking-related system calls restricted to Unix-domain sockets.

• TAME_DNSPATH is meant to allow hostname lookups; it gives access to a few system calls like socket(), but only after the program successfully opens /etc/resolv.conf. So the kernel has to track whether a few "special" files like resolv.conf have been opened during the lifetime of the tamed process.

• TAME_GETPW enables the read-only opening of a few specific files needed for getpwnam() and related functions. It will also turn on TAME_INET if the program succeeds in opening /var/run/ypbind.lock.

• TAME_CMSG allows file descriptors to be passed with sendmsg() and recvmsg().

• TAME_IOCTL turns on a few specific, terminal-related ioctl() commands.

• TAME_PROC allows access to fork(), kill(), and related process-management system calls.

A process may make multiple calls to tame(), but it can only restrict its current capabilities. Once a particular flag has been cleared, it cannot be set again.

The patch includes changes to a number of OpenBSD utilities. The cat command is restricted to TAME_MALLOC and TAME_RPATH, for example; never again will cat be able to run amok on the net. The ping command gets access to the net, instead, but loses the ability to access the filesystem. And so on.

This system call has a number of features that may look a bit strange to developers used to Linux. It encodes quite a bit of policy in the kernel, including where the password database is stored and the use of Yellow Pages/NIS; one would grep in vain for ypbind.lock in the Linux kernel source. tame() may seem limited in the range of restrictions that it can apply to a process; it will almost certainly allow more than what is strictly needed in most cases. It thus lacks the flexibility that Linux developers typically like to see.

On the other hand, using tame(), it was evidently possible to add restrictions to a fair number of system commands with a relatively small amount of work and little code. Writing ad hoc BPF programs or SELinux policies to accomplish the same thing would have taken quite a bit longer and would have been more error-prone. tame(), thus, looks like a way to add another layer of defense to a program in a quick and standardized way; as such, it may, in the end, be used more than something like seccomp().

If the tame() interface proves to be successful in the BSD world, there is an interesting possibility on the Linux side: it should be possible to completely implement that functionality in user space using the seccomp() feature (though it would probably be necessary to merge one of the patches adding extended BPF functionality to seccomp()). We would then have the simple interface for situations where it is adequate while still being able to write more flexible filter policies where they are indicated. It could be the best of both worlds.

The first step, though, would probably be to let the OpenBSD project explore this space and see what kind of results it gets. The ability to try out different models is one of the strengths that comes from having competing kernels out there. The ability to quickly copy that work is, instead, an advantage that comes from free software. If this approach to attack-surface reduction works out, we in the Linux world may, too, be able to tame() our cat in the future.

Comments (73 posted)

Privacy questions for Iceweasel

The Iceweasel web browser is Debian's rebranded Firefox, which came about when Mozilla's trademark policy was deemed too onerous by the distribution. Some recent complaints about Iceweasel making a number of HTTP requests on first startup that reduce privacy were heard; they were initially assumed to be coming from the Firefox code. That turned out to be a wrong guess—in fact, it was some changes made by Debian to remove possibly non-free icons that led to the requests.

The issue came up in a thread that started with a post from "lumin" that pondered on purity in the free-software world, especially with respect to non-free blobs and the need to use them at times. The thread meandered a ways, touching on the Chromium binary blob download we looked at in June, before Jakub Wilk posted the results of an experiment he ran with Iceweasel. In it, he noted that Iceweasel makes a bunch of HTTP requests when it starts up for the first time—even before the user types in their first URL.

Wilk's experiment found that the browser requests favicon.ico files from various search sites (eBay, Google, Yahoo, Amazon, Wikipedia)—requesting some of them multiple times. It also makes a few HTTP requests to Mozilla sites (including one POST) and a POST to Google's Safe Browsing site, followed by dozens of GET requests to that site. His concluding line from that message was one of our security quotes of the week: "So nothing serious here. It's just casually violating your privacy."

The Safe Browsing traffic caught the attention of Bas Wijnen, who was concerned that Google could be using it to track users. It turns out that Firefox (thus Iceweasel) implements a version of the Safe Browsing API that only sends a 32-bit prefix of the SHA-256 hash of the URL in question. It gets back a list of hashes that have that prefix and are considered to be dodgy sites. Google never gets the full URL, as that comparison is handled on the client side. Even if it were to get the full hash, reversing SHA-256 to extract the actual URL is, effectively, impossible, though the search provider could (and well might) have some giant lookup table of all the URLs it knows about and their hashes.

So, other than notifying Google that Iceweasel is being used, those requests are not leaking information. That is a privacy hazard, but one that can't be avoided if Safe Browsing is enabled. Wijnen's complaint that there was no way to turn it off was answered by Debian Iceweasel package maintainer Mike Hommey, who reported that there were settings in the Security section ("Block reported attack sites" and "Block reported web forgeries") to enable and disable the feature.

But Nikolaus Rath questioned the whole idea that any of the Iceweasel traffic was truly unexpected or unwarranted:

The Mozilla URLs were not really discussed in the thread. The POST is evidently a way for the browser to retrieve what country it is being run from, while the two GET requests are in support of the tiling feature for new tabs. They appear to simply be part of the Firefox feature set, so any privacy complaints may be sort of swept away as in Rath's admonishment above.

Ian Jackson and others focused on the download of the favicon.ico files. In effect, Iceweasel notifies the search providers whenever a user starts up the browser for the first time, even if the user doesn't want or need a particular provider—or is only opening the browser to look at a local file. But those icons are used in the search box to indicate where to send a search query, so they are needed from the initial browser startup.

Firefox from Mozilla (and in other Linux distributions) simply ships the icons with the browser. However, it is unclear whether those icons can legally be modified by users, so they were removed from the Iceweasel package because they do not conform to the Debian Free Software Guidelines (DFSG). Instead, Iceweasel was set up to download them at first run, as Hommey reported.

Jackson was surprised to hear that Debian had deliberately made a change like this that reduced users' privacy but didn't really change the legal situation for users:

He suggested that either an exception be made for the icons, since their distribution is permitted by upstream (Mozilla), or that the icons be eliminated entirely. For his part, Hommey is not even convinced that there is a problem with the icons vis a vis the DFSG. Mozilla has said that the icons are compatible with the Mozilla Public License (MPL) that Firefox is distributed under. So he announced a change in plans:

Since it "seems to be a minor issue", Debian project leader (DPL) Neil McGovern is not interested in referring it to the lawyers unless someone wanted to push that. Disabling the icons entirely, as suggested by some, is not really an option; Hommey demonstrated that with a screen shot. Another option would be to create new icons, perhaps some that are less ambiguous about who the search provider is, "so you don't have to guess whose icon is a briefcase", as Don Armstrong put it. He posted a proof of concept shell script that created icons with the initial six characters of the name of the provider.

In the end, it would seem that the icon files will simply be added back into the Iceweasel package as Hommey stated. That is certainly the simplest, least privacy-invading solution. While it may violate the DFSG, though that is not universally agreed upon, the alternative of downloading the icons seems worse to many. This episode is a reminder that preserving privacy in today's web world is a tricky prospect, but also, perhaps, a nod toward the realities that a 16x16 pixel "image" may not be the right place to enforce a—quite sensible, overall—line in the sand. It is still possible that someone will take up the issue and perhaps keep those icons out of the Iceweasel package (replacing them with Armstrong's or something similar), though it seems like that might be time better spent on other battles.

Comments (11 posted)

Distribution quote of the week

Comments (2 posted)

Red Hat Enterprise Linux 6.7 released

Comments (27 posted)

Rkt 0.7.0 released

Comments (22 posted)

Debian Installer Stretch Alpha 1

Full Story (comments: none)

Improving ARM64 support in Debian

Full Story (comments: none)

GCC 5 for 15.10 (wily) on July 31

Full Story (comments: none)

Distribution newsletters

• Debian Project News (July 21)
• DistroWatch Weekly, Issue 619 (July 20)
• Ubuntu Weekly Newsletter, Issue 426 (July 19)

Comments (none posted)

The New Solus: Putting the Pieces Together Again (LinuxInsider)

Comments (none posted)

A beta release and a new license for Mailpile

Mailpile is a free-software webmail client that has been designed from the start to provide top-notch security and privacy features. The project recently released its third beta, which improves setup and encryption-key discovery as well as streamlining the user interface. In addition, the project recently held a public discussion about the license under which Mailpile should be released, eventually settling on the Affero GPL (AGPL).

To recap, Mailpile offers a somewhat unusual take on the traditional email program. It is designed to run as a web application (thus making it cross platform with a minimum of fuss), but it is a single-user application and is designed primarily to run on the local machine. Because each user runs a separate copy of Mailpile, it offers isolation. Also noteworthy is that Mailpile is a mail user agent (MUA), so a separate server is required to actually send and receive mail. But, because Mailpile downloads all messages locally, it can function offline and (at least in theory) offer better safeguards against security breaches of the server.

The project also puts a strong emphasis on integrating cryptography into the webmail experience. We examined the first beta release in September 2014, at which point its GnuPG integration was rather awkward in places. The team made a second beta release in January 2015, but withdrew it from the site in March after beta testers complained about IMAP and GnuPG issues (although the release is still available from the project's GitHub repository). The team subsequently scaled back their planned feature set.

Beta three

The third beta was pushed out on July 19, and is available in a source-code bundle for Linux and as installers for Windows and Mac OS X. Mailpile is a Python 2.7 application, though, so installation for the source is trivial. The dependencies are fairly standard and the required Python packages can be installed through pip. Of the other dependencies, the only worthy of note is that Mailpile still uses the GnuPG 1.x branch.

Among the significant changes in beta 3 is a simplified setup and first-run configuration process. All that the user has to do is select a language and pick a password that will be used to decrypt the local mail storage (i.e., a separate password from those needed to access email accounts). Email account setup has been simplified as well; the program attempts to detect the correct IMAP and POP settings based on the email address entered, and the setup process includes a GnuPG key-generation step.

The server-settings-detection step uses Mozilla's ISP database, so it should be as reliable as Thunderbird's detection (and similarly limited to public email providers). Local mailboxes are also supported by the account-setup tool, so users who already get their mail in mbox or Maildir format can make use of this release as well.

There are even more noticeable changes to the mail-reading interface—most are, again, simplifications. Previous Mailpile releases included a sometimes-confusing array of tagging and contact-management features that, as the release announcement put it, "seemed like good ideas but never quite worked." For instance, there used to be three separate buttons linked to the user's tag collection, plus separate "flag" and "favorite" features. The resulting interface is easier to navigate and the clutter is not missed.

The new release also significantly improves the multiple-email-account experience, which was among the complaints with previous releases. There is now a "home page" that shows a summary of the configured accounts. Last but certainly not least, the GnuPG support now includes key discovery. The program will attempt to detect which email recipients use encryption or PGP signatures and try to fetch the corresponding keys from public keyservers. This is in addition to manually importing keys, which was already supported.

The program still has a ways to go before it is ready for production usage. Most notably, it lacks the security hardening required to make it usable from a remote server. Despite Mailpile's original design as a local client, this is evidently a feature that users repeatedly ask for.

License one

The other significant development from the past month is that the Mailpile team has finally decided on the license under which it will make releases. This is not an insignificant choice for any project, and the team took the unusual step of asking its users to weigh in on the choice.

In May, project lead Bjarni Einarsson posted an appeal to the community asking for its input. The options presented were the AGPL (version 3) and the Apache 2.0 license—which, at least in some respects, constitute opposite ends of the free-software license spectrum. Apache is renowned for its permissiveness, while AGPL is held up for doing the most to preserve copyleft. The preceding alpha and beta releases of Mailpile had been offered under both licenses while the team debated their merits.

In the initial post, Einarsson reiterated the usual arguments heard from proponents of each side: the risk of "marginalizing" the project by choosing the strong-copyleft AGPL, versus the risk of proprietary Mailpile forks by choosing the Apache license. In June, he summarized the feedback the team had received, quoting several blog posts and emails. Supporters of the project (in the financial sense) were encouraged to vote on the web site.

On July 2, the project announced the final decision: AGPL. Einarsson noted that AGPL "won" on a straight vote tally, albeit by a slim margin, and that Apache won when the results were adjusted by the dollar amounts of the supporters. But that latter result was skewed significantly by one voter's large donation; without it, AGPL won the dollar-adjusted vote by a slim margin, too.

Ultimately, Einarsson said, the slim margins and low turnout concerned him greatly, but he chose to go with the AGPL because he felt it was better aligned with the project's goals:

For now, it has not been announced whether or not additional beta releases of Mailpile will be made. For the time being, Einarsson plans to make incremental updates to the current beta release every two weeks or so, but he is also re-examining the roadmap. The final release could still take a while, but Mailpile has made clear progress in recent months, and now has a clear licensing plan going forward.

Comments (20 posted)

Quotes of the week

Comments (none posted)

dgit 1.0 released

Full Story (comments: 2)

Kubernetes 1.0 released

Google has released version 1.0 of its container-orchestration system Kubernetes. As the announcement explains, the 1.0 milestone designates Kubernetes as "production ready" for deploying and managing a variety of container workloads, coordinating related containers in "pods," and managing live clusters.

Comments (none posted)

Mozilla Winter of Security is back

At the Mozilla Blog, Julien Vehent announces that Mozilla will be conducting a second round of its "Winter of Security" mentoring program. Aimed at college students, the program allows participants to work on security-related free software for university credit, with guidance provided by Mozilla project members. This year's targeted project list includes some high-profile projects like Let's Encrypt and Mozilla's digital forensics tool MiG. Applications are due August 15.

Comments (none posted)

PyQt v5.5 Released

PyQt 5.5 has been released. PyQt is a set of Python bindings for Qt; the 5.5 release updates the bindings for Qt 5.5 compatibility. This includes support for the new QtLocation and QtNfc modules. Python 2.6, 2.7, and 3 are all supported.

Full Story (comments: none)

Synfig Studio 1.0 released

Version 1.0 of the 2D vector-animation suite Synfig Studio has been released. The latest release is actually numbered 1.0.1, due to a packaging problem with the original 1.0.0 upload. The list of changes includes bugfixes for working with animation keyframes, several new icons, and UI improvements when selecting bounding boxes on the canvas.

Comments (none posted)

Development newsletters from the past week

• What's cooking in git.git (July 20)
• What's cooking in git.git (July 21)
• LLVM Weekly (July 20)
• OCaml Weekly News (July 21)
• OpenStack Community Weekly Newsletter (July 17)
• Perl Weekly (July 20)
• PostgreSQL Weekly News (July 19)
• Python Weekly (July 16; issue #200)
• Ruby Weekly (July 16)
• This Week in Rust (July 20)
• Tor Weekly News (July 22)
• Wikimedia Tech News (July 20)

Comments (none posted)

Calculating the "truck factor" for GitHub projects

Comments (27 posted)

Webber: Why I Am Pro-GPL

At his blog, Chris Webber has written a response to Shane Curcuru of the Apache Software Foundation, who delivered a "Why I don’t use the GPL" lightning talk at this year's OSCON. In particular, Webber takes issue with Curcuru's assertion that he rejects the GPL because he "cares about the users." Writes Webber:

Webber proceeds to respond to several arguments raised in the talk, and concludes: "I have heard a mantra many times over the last number of years to "give away everything but your secret sauce" when it comes to software development. But I say to you, if you really care about user freedom: give away your secret sauce."

Comments (4 posted)

"Cloud Native Computing Foundation" launched

Comments (4 posted)

Open Document Format (ODF) 1.2 published as International Standard

Full Story (comments: none)

2nd Annual Web Audio Conference - submissions now open

Full Story (comments: none)

CFP Deadlines: July 23, 2015 to September 21, 2015

If the CFP deadline for your event does not appear here, please tell us about it.

Events: July 23, 2015 to September 21, 2015

If your event does not appear here, please tell us about it.