LWN.net Weekly Edition for April 23, 2015

Plotting tools for networks, part II

In part one of this article, we took a look at Dia, which provides a GUI for creating flowcharts and network diagrams, and at Graphviz, a toolbox and declarative language for graphs. We conclude here by introducing a package used with LaTeX to allow you to embed graphs seamlessly into documents, followed by a Python library for exploring the properties of networks.

TikZ: LaTeX can draw

In our look at gnuplot earlier in this series, we saw how gnuplot could cooperate with LaTeX to help produce a document where the plots and the text blend together seamlessly. TikZ is another option: a LaTeX package that provides a language for describing diagrams. It can be used with plain TeX in addition to LaTeX; this TUGboat article [PDF] provides a nice overview. TikZ is a front end, or syntax layer, for LaTeX's portable graphics format (PGF), and is included with PGF packages. PGF has other front ends useful for other purposes and it can also be used directly.

TikZ employs an intuitive syntax and can define any conceivable diagram, from electrical circuits to calendars. It has grown into a vast ecosystem for graphics—including its own set of contributed libraries—that is comparable to LaTeX itself. The PDF manual that comes with the current version, written by TikZ's original creator, is 1165 pages long.

For these examples, we used the latest stable version of TikZ, version 3.0.0, which is the first version to include the graph-drawing facilities we see below. Installations of TeX Live from distribution package managers might come with somewhat older versions of PGF and TikZ; if you need a more recent version, you can find it at the PGF site. In my case, installing a newer TikZ (including the massive documentation) was a simple as downloading an 11 MB zip file, dropping it into my ~/texmf directory, and unzipping. It has worked flawlessly with my TeX Live installation from 2012.

TikZ's graph-drawing language is inspired by Graphviz's dot (see part one for more detail). Here is a minimal LaTeX document that produces a simple diagram showing the classification of a few musical instruments. It can be processed by any of the PDF LaTeX engines:

    \documentclass{minimal}
    \usepackage{tikz}
    \usetikzlibrary{graphs,quotes}
    \begin{document}

    \tikz \graph [nodes = {rectangle, draw},
                  circular placement,
                  radius = 2cm] {
       percussion/"\bfseries{percussion}" [blue, xshift = -5cm] ->
          piano [xshift = -2cm];
       strings/"\bfseries{strings}" [xshift = 2cm] --
          [dotted, bend right, thick] piano;
       strings -> guitar;
       guitar ->[red, "ancestor"] chitarra;
       strings -> violin;
       strings -> viola [yshift = 1cm];
       strings -> cello [xshift = -2cm];
       percussion -> xylophone [xshift = -1cm];
       percussion -> timpani [xshift = -3cm, yshift = 1cm];
    };

    \end{document}

To include TikZ diagrams, we need to import the tikz package, and following that, any TikZ libraries that we plan to use. The graphs library defines the convenient, dot-like syntax for the graph-theory diagrams that we're mainly concerned with here. The quotes library allows for more flexible printing of the labels for our nodes; we need it here because we would like to use some font commands in our node labels.

The actual drawing starts with the \tikz command. The syntax is almost self-explanatory. Options are specified inside of square brackets. The options immediately following the \graph command apply to the entire diagram. The nodes and their connections are listed inside a pair of curly brackets, with each connection punctuated by a semicolon.

The percussion/"\bfseries{percussion}" syntax defines a node that will be referred to as "percussion", but is printed with the label that's between the quotation marks. This shows how we can use LaTeX commands to specify how labels appear in the diagram. Connections between nodes are shown as "->" for directed edges (arrows) and "--" for undirected edges. Other options are listed in square brackets following the element to which they are to be applied.

Notice in particular the need to make liberal use of the xshift and yshift options, which do the expected: adjust the position of the node in the horizontal and vertical directions, respectively. The options we've used for this diagram only allow TikZ to use a primitive layout algorithm that, without these shift directives, lead to crowding and overlap. With some hand tuning, however, we have an attractive diagram that, moreover, will use the same fonts and line widths as the rest of the document.

In our look at Graphviz, we discussed that project's assortment of layout engines, and how they could be applied to the same logical graph, defined in the dot language, to produce different visual representations. TikZ has a similar facility, with several node-placement algorithms to choose from. The algorithms are implemented in the Lua scripting language, in order to leverage LuaTeX, a version of pdfTeX extended with Lua.

In fact, TikZ's graph-layout algorithms are one of the major success stories for LuaTeX, and an excellent example of the kind of thing that it makes possible. Here is the same diagram, invoking the "tree" layout algorithm:

    \documentclass{minimal}
    \usepackage{tikz}
    \usetikzlibrary{graphs,graphdrawing,quotes}
    \usegdlibrary{trees}
    \begin{document}

    \tikz \graph [tree layout, nodes = {rectangle, draw},
                               circular placement,
                               radius = 2cm] {
       percussion/"\bfseries{percussion}" [blue] -> piano;
       strings/"\bfseries{strings}" -- [dotted, bend right, thick] piano;
       strings -> guitar;
       guitar ->[red, "ancestor"] chitarra ;
       strings -> {violin, viola, cello};
       percussion -> xylophone;
       percussion -> timpani;

    };
    \end{document}

In this example, we import a new TikZ library called graphdrawing; this and the line following it load the "tree" algorithm. The tree layout option to the graph invokes the layout on our diagram. We've omitted all the manual shifts, allowing the automatic procedure to keep the nodes properly spaced. The result, shown here, is quite serviceable.

TikZ also includes a force-directed layout algorithm, similar to the fdp engine in Graphviz. The idea behind these algorithms, roughly speaking, is to imagine that the nodes are connected by springs, then to search for a configuration that minimizes the total force, or potential energy, of the arrangement. The result is influenced by the starting position used in the search, which in TikZ is controlled by a "random seed":

    \documentclass{minimal}
    \usepackage{tikz}
    \usetikzlibrary{graphs,graphdrawing,quotes}
    \usegdlibrary{force}
    \begin{document}

    \tikz \graph [random seed = 1,
                  spring layout,
                  node distance = 1.5cm,
                  nodes = {rectangle, draw}] {
       percussion/"\bfseries{percussion}" [blue] -> piano;
       strings/"\bfseries{strings}" -- [dotted, bend right, thick] piano;
       strings -> guitar;
       guitar ->[red, "ancestor"] chitarra ;
       strings -> {violin, viola, cello};
       percussion -> xylophone;
       percussion -> timpani;

    };
    \end{document}

Again, the results are pretty good, with no manual tweaking required:

NetworkX

The three packages we've discussed thus far provide three different approaches to drawing network diagrams. We've discovered that the same network can be laid out in many different ways, reinforcing the idea that a network has an abstract identity apart from any particular visual representation.

The mathematical theory of networks has led to insights about the spread of disease, the behavior of communications grids, the organization of biochemical pathways, and much more. NetworkX is a Python library for the study of these mathematical objects. Let's dive in to show how you can use it to interactively explore the properties of a simple network.

NetworkX can be installed using your distribution's package manager or with the Python package manager pip. Since it relies on either matplotlib or Graphviz for drawing, we need to have one of these installed as well if we want pictures. Note, however, that NetworkX's drawing routines are not yet ready for Python 3, but you can export the graphs in dot (or several other) formats for use with other drawing tools.

In the examples below we use the matplotlib interface. Once everything is installed, we can start the Python interactive prompt and import the libraries (in place of the second import, you can use IPython with the PyLab interface):

    >>> import networkx as nx
    >>> import matplotlib.pyplot as plt

We now need a network to explore. We could build one by adding nodes and edges one at a time, but NetworkX can read data in many different formats and parse it into its internal graph representation. A standard format for network data is the Graph Modelling Language (GML) text format. There are several public repositories of interesting network data formatted in GML; from one of these we borrowed a graph of the relationships between the characters in the novel Les Miserables.

To read the file and create a graph object from it, we merely need to say:

    >>> G = nx.read_gml('lesmis.gml', relabel=True)

The relabel argument tells NetworkX to use the "labels" from the file (which in this case are the characters' names) to label the nodes rather than their IDs, which are numerical tags that are also present in the GML file.

The following line will create a default drawing of the graph:

    >>> nx.draw_networkx(G, alpha = 0.5)

The alpha parameter sets the opacity of the circles representing the nodes. Depending on what version of the Python read–eval–print loop (REPL) you are using, and how it was invoked, the graph window may pop up now, or you may need to type plt.show().

As you may have anticipated from following the previous examples, the default graph is crowded, with a lot of overlapping nodes. One of the many operations on graphs that NetworkX can make is to calculate a network's "k-core", which is the largest sub-graph where all the nodes have at least k connections:

    >>> gk = nx.k_core(G, 8)

Instead of drawing this immediately, let's calculate the positions of the nodes using one of the available drawing routines, and save them:

    >>> pos = nx.spring_layout(gk)

NetworkX can calculate many interesting things about networks. What if we want to know the shortest way to get from one character to another, by traversing the network of relationships?

    >>> gs = nx.shortest_path(gk, source = 'Montparnasse', target = 'Joly')

This gives us the solution in the form of a list of nodes between the two characters:

    [u'Montparnasse', u'Gavroche', u'Joly']

This list is now stored in gs. NetworkX can also calculate shortest paths while considering weights assigned to the edges. For example, if we have a network of flights between airports with costs assigned to the connections, the shortest path would be the cheapest ticket between a given origin and destination.

The GML file assigns a "value" to each edge based on how frequently each pair of characters appear on the same page. To calculate a shortest path taking these values into account, we say:

    >>> gn = nx.shortest_path(gk, source = 'Montparnasse', target = 'Joly',
    >>>                       weight = 'value')

The list of nodes in gn is now:

    [u'Montparnasse', u'Eponine', u'Mabeuf', u'Joly']

In order to draw a picture of the calculated shortest paths, we need to turn that list of nodes into a list of edges, which is simply a list of tuples for each hop:

    >>> gs_list = []
    >>> for i in range(len(gs)-1):
    >>>     gs_list.append((gs[i], gs[i+1]))

Some of the drawing commands accept a list of edges; we'll draw this path in a distinct color and thickness:

    >>> nx.draw_networkx_edges(gk, pos = pos, edgelist = gs_list,
    >>>                        edge_color = 'blue', width = 4)

Here we've also used the pre-calculated node positions we created for the k-core, so our edges are aligned with the positions of the entire group. We need to do this because NetworkX will apply its layout algorithms with a different random seed each time (this is also why your drawings may look different from the ones shown here).

We now repeat these steps for the weighted path, using gn and an edge_color of red. The final picture emerges when we overlay the k-core, which might be considered the novel's in-group:

    >>> nx.draw_networkx(gk, pos=pos, alpha = 0.3)

From this picture you can see that an unweighted shortest path simply minimizes the number of hops; however, when taking edge weights into consideration, a solution with a larger number of hops may have a smaller total cost.

NetworkX contains a large number of routines, besides the two illustrated here, for calculating properties of networks. It's a powerful and mature library both for mathematical research and practical work.

Conclusion

Electronic circuits, family trees, transit maps, and social networks are all examples of graphs, the subject of a distinct branch of mathematics that began as a collection of recreational puzzles and has found a recent resurgence in the interdisciplinary science of networks. Graph theory is unusual as a modern branch of math in that it can be approached even at an elementary level and yet presents many deep and interesting theorems and practical applications.

Since these applications cross over into so many different domains, there is a wide variety of software that deals with graphs from various angles. In our look at network, flowchart, and graph software we've offered an overview of several free and open-source tools that should give you an idea of where to look to address your particular needs, all the way from drawing simple flowcharts up to analyzing complex networks.

Comments (none posted)

Python on mobile systems

A session "led" by Russell Keith-Magee at the 2015 Python Language Summit was actually a video he created about mobile Python. He has been a Django core team member since 2006 and is the co-founder and CTO of TradesCloud, which has led him to look into Python's role in the mobile space. His video was meant to provide an overview of the mobile development landscape, present the current state of Python for mobile devices, and to explain the decisions he believes the Python project needs to make in order to fully support mobile development.

From his perspective, the only sensible meaning for "mobile Python" is for it to mean "embedded Python"—the language bundled into a mobile app. There may someday be a market for the full Unix Python experience on an iPhone or other mobile device, but there isn't today. So by his definition, developers will be able to embed Python into a mobile app so that all or parts of the app can be written in the language.

In terms of mobile operating systems, there are quite few out there. His talk will be focusing on "the big two" (iOS and Android), he said. Not only are those two the bulk of the market, but they are also the platforms where most of the mobile Python work has been done so far.

For iOS, the toolchain is based on Clang. The binaries and libraries for iOS are "fat", as they actually contain the executables and object files for multiple architectures. Those binaries are created via multiple passes of the compiler, then merged together into a fat file. CPython can be relatively easily built as an embedded library in that environment. Native services on the platform can be accessed by bridging through the iOS Objective C runtime.

Android is quite different, with a runtime that is Java-based. You can access native services from C through the Java Native Interface (JNI), but there are some pretty serious limitations to that approach, especially with respect to reference counting (for garbage collection). It can be done, Keith-Magee said, but it is difficult. There are two possible approaches to supporting Python on Android: either using Jython-compiled bytecode or building the standard interpreter (i.e. CPython) as an embedded library. The latter is what has seen the most attention.

There are patches available for Python 3.4, either as patch sets or as meta-build scripts that grab the latest Python, patch it, and then build it. Kivy has gotten Python apps using its toolkit into both the iOS and Google app stores that way. The current focus of the mobile Python community is to get as many of those patches upstream as possible, he said.

To that end, there are issues in the Python bug tracker for adding support for iOS and for Android. In both cases, patches for Python 3.4.2 are available in the bug entries. The Python test suite is only partially working; just some simple smoke tests as well as tests to access data from C extensions using ctypes can be run. Both of these efforts are currently awaiting fixes to the ARM support in libffi.

The biggest outstanding technical issue at this point is the test suite and Buildbot integration, he said. The next step is to be able to build the test suite as an app, push it to the device, run it there, and retrieve the test results. Scripts need to be written to do that, but there is also a problem of hardware availability. Either someone needs to buy a bunch of devices and hook them up to a system somewhere or the community could engage a company that provides a service with many different mobile platforms for building and testing.

One decision that needs to be made by the Python core developers is how the platform identifies itself via the sys.platform variable. There are two schools of thought. One is that the platform should be "ios" or "android" as appropriate, while the other is that it should be "linux" for android (and presumably something else for iOS, but he didn't say). At least notionally, programs for other Linux distributions could run on Android, he said, which is why some advocate for the latter.

Building fat binaries/libraries is currently done with a meta-Makefile that runs configure and the standard Makefile multiple times, then merges the resulting targets into the fat files. That work could be moved into the standard Makefile instead, though there are still some difficulties because configure must be run each time for different architectures.

The current approach is to build all of CPython into a single library because Apple does not allow dynamic linking for iOS apps. That makes for large app file sizes, but that size can be cut down by only putting the modules needed by the app into the library. The patches for iOS provide a mechanism to do that, but it may be a desirable feature for more than just that platform, he said.

Apps will need to access native system services to do things like draw widgets on the screen. There are a number of different projects that allow Python to bridge to the native runtimes, but Keith-Magee wondered if some kind of bridging was in-scope for the standard library. If other services like geolocation or access to the accelerometer are to be provided by a mobile framework in the standard library, some form of bridging would be needed. After the video, several attendees seemed pretty adamant that those kinds of features were not at all in-scope for the standard library—at least in their minds.

There is also a question of what gets shipped. Apps are self-contained, so the project can't just ship a "Python app" that other apps use. For Android, the project could build a library for each platform; for iOS, a single fat library could be created. The problem is that apps are likely to want customized builds that are stripped down to only what that app needs. There is also the question of which CPU versions and platform versions will be supported by the project. For Android, the platform problem is particularly acute because there are so many different versions of Android running on devices. In fact, Android 2.3 ("Gingerbread") is shipping on brand new devices being made today, he said.

Third-party modules also raise questions. Pure Python modules are easy—just put the code somewhere on the PYTHONPATH—but C extensions raise some other issues. They will need to be built into the library somehow; will there be a standard mechanism to do so? Ethan Furman has volunteered to shepherd the Android patches moving forward. As far as he is aware, there is no one currently doing so for iOS, Keith-Magee said.

Many of the issues he raised could be deferred for a while, but there are a few that really need to be resolved before any forward progress can be made. Changes to the build system to support the mobile operating systems, as well as the platform identification issue, are the most pressing, he said.

Comments (1 posted)

Adding Requests to the standard library

The Python Requests module is a popular alternative to the standard library's HTTP handling (e.g. urllib2). Kenneth Reitz, who is the "benevolent dictator for life" (BDFL) for Requests, came to the Python Language Summit to discuss the possibility of adding the module to the standard library. It is oft-requested, but is a bit controversial, he said. In addition, he wanted to use his slot as a forum to discuss the criteria for adding things to the standard library.

Requests has security as its top priority. It focuses on using industry best practices for things like SSL/TLS, connection pooling, encodings, headers, and so on. It removes a significant amount of the complexity for writing programs that interact with the web.

It is also the most popular package on the Python Package Index (PyPI, though Reitz used its other name: The CheeseShop), having been downloaded some 42 million times. That is more downloads than either setuptools or virtualenv. That popularity leads to frequent suggestions that Requests be added to the standard library.

The development of Requests is mostly done by Reitz and two core contributors. It has been feature-frozen for the last two years. It has a stable API and a stable development community, he said.

There are two major dependencies for Requests that would also have to be added if Requests moved into the standard library. One is chardet, which is an encoding detector that is based on Mozilla's character-recognition algorithm. It is quite useful when dealing with Unicode because servers can't always be trusted to specify the encoding correctly. It would be a good addition to the standard library regardless of what happens with Requests, he said. The other dependency is urllib3, which provides thread-safe connection pooling, file posting, and more. It is under active development and receives updates frequently.

The Requests project sees the module as "critical infrastructure for the Python community", Reitz said. For example, it is used by the pip Python package installation tool.

There are a number of arguments for inclusion into the standard library. If the Python community wants to provide libraries that embody best practices, then adding Requests would be "the right thing to do". From a sustainability standpoint, having Requests in the standard library would make it easier to get funding for the core contributors. Also, chardet would be good from a "batteries included" standpoint.

Reitz also relayed some observations that might make it difficult for Requests to be included. To start with, it comes with its own "carefully curated" (and frequently updated) bundle of certificates from certificate authorities (CAs) for SSL/TLS verification, while Python relies on the system CA bundle. Relying on the system certificate bundle would likely reduce the security of the library, he said.

Beyond that, the HTTP specifications and recommended usage change significantly over time; the Requests module keeps up with those changes. There are also situations where the project has turned around a release for a security fix in 12 hours, which would be difficult or impossible to do if it were included in the standard library.

The "biggest pitch" for Requests is that it is better than the parts of the standard library that cover the same ground. Pulling Requests into the standard library properly would require a bunch of work to integrate it and replace pieces of the standard library with parts of Requests. The project would also lose the ability for fast turnaround changes based on security problems or specification/usage changes.

He also wondered about the goals of the standard library. Now that Python ships with pip, is inclusion in the standard library really needed for modules like Requests? The "official stance" of the Requests project is that it is critical infrastructure, as he said earlier, but also that it is too critical to be included in the standard library.

That conclusion seemed largely agreeable to those present. One attendee said they would not be happy if they had to update Python to get a new Requests. Nick Coghlan noted that network security needs make the update cycle for Requests quite different from that of Python as a whole. It is also impossible to maintain network security for a release that has a "no new features" policy (e.g. Python 2.7).

There needs to be a way for new people to get the recommendations to use "external" modules like Requests, Łukasz Langa said. But there is a problem for some users (and companies), though, because they don't want to install additional, third-party dependencies, Thomas Wouters said. Requests is in a somewhat different category, though, since it is already installed for pip, Coghlan said.

Someone suggested that there be a set of modules that are vetted and endorsed by the core developers or the Python Software Foundation. Langa wondered what would happen if the maintenance stopped for an endorsed package, however. He suggested that the documentation get changed so that there is a section on deprecated modules and their suggested replacements.

There were some other concerns expressed. Glyph Lefkowitz argued that one of the big hurdles for users adopting Requests and other PyPI modules is the command line. Adding some sort of user interface to pip would help with that, he said.

Alex Gaynor noted that Python 3.4 added the asyncio module, but that Requests does not support it. He wondered how Requests could even be considered for inclusion in the standard library without asynchronous support using asyncio. Brett Cannon pointed out that there is a need for an informational PEP that describes the goals of the standard library.

Larry Hastings kind of summed up the session when he said that "batteries included may not make so much sense for everything anymore". Certainly the feeling in the room seemed to indicate that there is a mismatch between the frequency of Python releases and the needs of some modules, which means their users are better served by remaining separate.

Comments (44 posted)

PyMetabiosis

Romain Guillebert, who is one of the PyPy developers, introduced his PyMetabiosis project at the 2015 Python Language Summit. The idea behind it is to make it easier for people to use alternative Python implementations (such as PyPy) by making C extensions available to them.

The biggest barrier to using alternative implementations is the availability of the C extensions, he said. PyPy and others cannot just support the C API because it is too closely tied to the CPython implementation details. For example, it assumes reference counting for garbage collection, which is known for being inefficient and for making it difficult to get rid of the global interpreter lock (GIL). PyPy doesn't use reference counting; it can emulate it, but that is slow.

So he has come up with a "hackish way to make C extensions work" for PyPy. PyMetabiosis embeds CPython into PyPy, which allows importing C extensions using CPython. PyPy wraps a proxy around a CPython PyObject that implements the needed methods, which call out to an extension via the C API.

PyMetabiosis allows a lot of Python extensions to be accessed from PyPy. For example, it allows sharing NumPy arrays between PyPy and CPython without any copying. It also did not take much code to implement PyMetabiosis; Guillebert said he spent about two weeks getting it working.

It is, unfortunately, still slow. It also makes PyPy dependent on CPython, which is not ideal. It doesn't work on Windows either. PyMetabiosis uses an environment variable that must point to a CPython virtual environment. Another limitation is that PyPy objects that are passed to CPython and get altered (e.g. lists) do not have those changes reflected in PyPy.

Ultimately, though, PyMetabiosis is just another slow compatibility layer that has different properties from other choices, he said. He would like to see an effort to incrementally improve the C extension API. He suggested looking at the APIs provided by other languages, such as those for Java (using the Java Native Interface), V8 JavaScript, Lua, or Julia, which are "way more" implementation independent.

Someone asked about "going the other way around" and embedding PyPy into CPython for use by performance-sensitive code. Often the numerical crunching done by NumPy and SciPy needs all of the performance boost it can get, while the rest of the code could be terribly written, but it doesn't really matter performance-wise. Guillebert agreed that both ways would work.

The current situation, with both a NumPy and a NumPyPy for the two different Python implementations, is "not a great long-term solution", an attendee said. Users really just want one version, but they need some path to get at PyPy's just-in-time (JIT) compiler. If there is an effort to rewrite or enhance the C API, providing that path should be considered.

Nick Coghlan said that the C Foreign Function Interface (CFFI) should be refactored so that it can serve as an extension module generator. CFFI is based on Lua's C API. Glyph Lefkowitz suggested that extension developers are often ignoring CFFI; when they actually try to use it, CFFI generally solves their problems, he said. The key is to get people to stop hand coding their C extensions, Coghlan said.

Comments (5 posted)

Jython Native Interface (JyNI)

In his session at the 2015 Python Language Summit, Stefan Richthofer described a project he has been working on for the last two years to enable Jython (Python on the Java virtual machine) to use Python's C extensions. It is called Jython Native Interface (JyNI) and it provides an adaption layer between Jython, which, like Java, uses the Java Native Interface (JNI) to access native code, and Python's C API.

Jython reimplements various built-in Python types (e.g. PyIntObject, PyStringObject, ...) in Java. The idea behind JyNI is to create a C-compatible view into these objects that stays in sync with them. The C API is "large and complex", which is one obstacle that needs to be surmounted. In addition, CPython uses macros in some places to access data objects; those macros cannot be switched out at runtime by dynamically loaded functions.

There are also a few minor obstacles to JyNI, including exception handling and the global interpreter lock (GIL), but the biggest problem area is garbage collection. Making it all work without leaking memory is challenging, he said.

There are three approaches that are being used to unify objects between Java and Python (i.e. to allow access as either a PyObject or a jobject). For Python object types that have Java versions available, a PyObject wrapper for the jobject is used. If there is no object of that type available in Java, the reverse is done (a jobject wraps a PyObject). The third option is used when macros are used to access the object in Python. It is a data structure that mirrors the object state between a PyObject and a jobject version. JyNI tries to avoid mirroring but it cannot be avoided for some kinds of objects.

CPython provides garbage collection for extensions, so some extensions may be relying on that for their memory management. Java does not provide any garbage collection to native code, so extensions must manage their own memory. Resolving that will, seemingly, be the biggest challenge for JyNI going forward.

Comments (2 posted)

Python installation options for Windows

Steve Dower took over the maintenance of the installer for Python on Windows a year ago, just after PyCon 2014. Since then, he has gotten a lot of feedback that the installer is not doing what is needed. He spoke at a 2015 Python Language Summit session to explain what the installer is doing and to describe some scenarios that it might support.

The Windows installer for Python 3.4 and earlier had a rather large security hole in that it allowed any user to modify the system Python installation. Dower has fixed that for the 3.5 installer by putting system-wide installations into a directory (Program Files) that can only be modified by the administrative user.

There are three different categories for installations that Dower identified. He wondered which the Python project should be delivering. The first type of installation is the "as if bundled" by the operating system. That is really a Windows-only question because the other operating systems' installers already do this. An important consideration is that adding modules to this type of installation must be governed by having administrative privileges.

There is also the "just for me" case, where Python is installed only for one user. That is an important use case and for Python 3.5 it is supported by the installer. The downside is that those installations don't get security and other updates from the system administrator.

The final installation scenario is one that is private to an application. The result would not make Python available to anything other than the application that was installed. In addition, no changes to the modules available would be allowed except via the application's update process. There could be a ZIP file created as part of the Python build process that would be stored on python.org. Applications (e.g. the Mercurial version control system) could then bundle a particular version of Python. Thomas Wouters noted that this scenario is not Windows-specific, as applications for other operating systems might also want that ability.

No real conclusions came out of the session. It was more informational, seemingly, to update the developers present on the installer and Dower's thinking about it.

Comments (none posted)

Python at Heroku

The final session of the day at the 2015 Python Language Summit was from Jacob Kaplan-Moss, who is a core contributor to the Django web framework and the director of security at cloud-application hosting service Heroku. He presented some numbers on how Python fits into the Heroku ecosystem, though he asked that attendees not share the numbers themselves. However, a general idea of Python's place at the company can be extracted from his talk.

Ruby is the most popular language used on Heroku, which is not a surprise since the other languages have only been supported on the platform for two years or so. JavaScript using Node.js is the second most popular, with Python in third place. Node.js is roughly half as popular as Ruby with "active developers", while Python has a quarter of the popularity of Ruby. PHP is next after Python; Java and Scala make up a fairly small portion of the languages used by Heroku's active developers.

A graph of the change in relative popularity (as compared to Ruby) showed Node.js "exploding", while Python is experiencing steady growth in popularity. Python is the second fastest growing language at Heroku after Node.js.

The list of most popular add-ons (packages that Heroku offers its Python developers) was not terribly surprising. The list reads something like a "who's who" of internet products and technologies: New Relic, logging, Redis, email, Memcached, MongoDB, and so on. He noted that Heroku users always get PostgreSQL, so it does not appear in the list.

Python 2.7 totally dominates the Python versions being used by Python applications in the last year, though it should be noted that it is the default version. Python 3.4 is next, but is smaller than 2.7 by a factor of ten. All other Python versions are less than one percent and the numbers for PyPy are "abysmal", which he found surprising. If you look at the numbers from just the last month, though, Python 3.4 has made some significant progress against 2.7. Part of that may be because the Django tutorial now uses Python 3 by default, he said.

In terms of Web Server Gateway Interface (WSGI) servers, Gunicorn completely dominates, while only Waitress has anything more than a small fraction of use by active Python applications. In terms of Python web frameworks, a bit less than two-thirds of active applications use Django and roughly one-third use Flask. That obviously doesn't leave much room for alternatives, with Tornado taking up the bulk of what's left.

It is important to note that these numbers are not general Python usage numbers, he said. The numbers come from "Python users doing web things", which definitely skews the results.

Comments (1 posted)