Development

Python context managers

If you have been programming for a while now, you might have dealt with issues related to databases, network sockets, files, and so on, which are often called resources. To understand what we mean by resources, think of anything that is both useful and in limited supply. The main feature of resources and, in particular, their limited availability is that once you finish using them they have to be returned back to the system. Failure to do so can result in various problems, one of which is a memory leak, meaning that memory is allocated but not freed. Context managers in Python can help with resource handling.

The underlying issue is what's called resource management. This includes any task that the programmer or the Python interpreter needs to carry out to make sure that any resource that has been used is returned back to the system. Let's look at an example to clarify this further. Say you have created a new database mydb, with one table Person, that contains three fields (columns) ID, Name, Age, as follows:

    import sqlite3

    connection = sqlite3.connect('mydb.db')

    connection.execute('''CREATE TABLE PERSON
                          (ID INT PRIMARY KEY NOT NULL,
                           NAME TEXT NOT NULL,
                           AGE INT NOT NULL);''')

    connection.close()

Assume that some exception occurred when running the execute() method. In this case, the database connection will never be closed, eventually leading to a memory leak. If a similar script was part of a function that gets called frequently (i.e. a query in a web server), the situation could be particularly problematic.

We can thus conclude that this wouldn't be the best way to implement this kind of function. A good solution to this is to handle the exceptions in order to close the database connection even when an exception has occurred. This could be done using the try/finally clause:

    try:
        connection.execute('''CREATE TABLE PERSON
                              (ID INT PRIMARY KEY NOT NULL,
                               NAME TEXT NOT NULL,
                               AGE INT NOT NULL);''')
    finally:
        connection.close()

Although the try/finally clause solves the problem, it still raises some questions, such as whether we need to make another outer try/finally clause to catch an exception that was raised by sqlite3.connect(). If we need to make such outer try/finally clauses, we will surely end up with messy code. There must be a better solution.

Context managers can be part of that solution. They wrap a block of code and ensure that specific operations are performed before and after it. In other words, a context manager will be responsible for a resource within the code block such that it ensures the resource is created when the block is entered, and cleaned up when the block is exited.

Context managers were first introduced more than ten years ago, in Python 2.5, by Guido van Rossum and Nick Coghlan. PEP 343 added a new keyword to the language, with, that enables context managers. Thus, when using the with statement, we are actually entering a context manager.

Context manager syntax

The with statement works much the same way as try/finally blocks, but with a cleaner syntax. Based on Python's documentation:

Using a context manager has the following syntax:

    with expression [as variable]:
        some_code

The expression produces a context manager object, while the variable refers to the object returned by the context manager's __enter__() method (more on this in the next section). Let's rewrite the above script using a context manager. It could look as follows:

    import sqlite3

    with sqlite3.connect('mydb.db') as connection:
        connection.execute('''CREATE TABLE PERSON
                              (ID INT PRIMARY KEY NOT NULL,
                               NAME TEXT NOT NULL,
                               AGE INT NOT NULL);''')

Inner workings

In the above context manager, the with statement will evaluate the sqlite3.connect() expression, resulting in an object known as a context manager that defines two special methods: __enter__() and __exit__(). The __enter__() method is called when a context manager is created in a with statement, returning a result that is assigned to the variable after the as keyword. The __exit__() method, on the other hand, is called when the context manager goes out of scope after the with statement. The object returned by the sqlite3.connect() function is a database connection object that implements the context manager protocol.

Notice that we don't need to use a finally block with context managers, since they are guaranteed to execute the __exit__() method. If we are working with files, for instance, the __exit__() method always guarantees that the file is closed if it was opened.

The prototypes for __enter__() and __exit__() are as follows:

    def __enter__(self):

    def __exit__(self, type, value, tb):

The __enter__() method runs immediately when the expression is evaluated. The __exit__() method is called with four arguments when the context manager goes out of scope: self, an exception type, an exception instance, and a traceback object. If there are no exceptions within the block, all the arguments (excluding self) are set to None.

Say we want to define a class where its instances act as context managers, we can do the following:

    class MyFile():
	def __init__(self, file_name, mode):
	    self.file_name = file_name
	    self.mode = mode

	def __enter__(self):
	    self.open_file = open(self.file_name, self.mode)
	    return self.open_file

	def __exit__(self, type, value, tb):
	    self.open_file.close()

Using the context manager, we can write:

    with MyFile('foo.txt', 'w') as file:
        file.write('foo')

The above context manager will simply create a file named foo.txt, and write the word foo to that file. When the with block is exited, the __exit__() method will ensure that the file gets closed.

Why use context managers?

Context managers are mainly used for managing resources. For instance, when we open a file we are consuming a resource called a file descriptor, that is an integer handle assigned by the operating system to refer to an I/O channel (i.e. file). Context managers make sure that any opened file is also closed, since there is a limit on the number of open files a process can have at one time. The command ulimit -n can show you the number of those files. In my case it is 256 files.

Trying to always remember adding a close() function after each open() is not an easy task, especially if the file was located in a complicated function that has multiple return paths or can raise exceptions. Thus, context managers guarantee that cleanups always happen.

Nested context managers

The with statement can have nested (multiple) context managers that can be separated by commas. Nested context managers can be written in the following different ways:

    with M() as m, N() as n:
        code

or:

    with M() as m:
        with N() as n:
            code

It is important to note here that the __exit__() functions for both context managers will run when the statement block exits.

contextlib module

The contextlib module contains utilities for creating context managers. Let's take an example that uses contextlib.closing(), which will ensure that the object passed to it is closed, to clarify that further. Let's say we want to open some web page and print the lines of that web page, we can do the following:

    import contextlib
    import urllib.request

    url = 'http://abder.io/'
    with contextlib.closing(urllib.request.urlopen(url)) as webpage:
	for line in webpage:
	    print(line)

Here, when the with statement runs, the contextlib.closing() helper creates a context manager on the fly to ensure that the URL object gets closed. It is important to note that using contextlib.closing() replaces the need to define both the __enter__() and __exit__() methods.

Conclusion

Context managers come in handy to manage resources in Python, because the __exit__() method is always guaranteed to execute and can clean ensure that those resources are cleaned up. The contextlib module contains helpers for creating context managers. It is worth considering reaching for the context manager tool when building your next Python application.

Comments (12 posted)

Development quotes of the week

— Tom Tromey

— Bill Sourour (Thanks to Paul Wise)

Comments (2 posted)

Firefox 50.0

Mozilla has released Firefox 50.0. This version features improved performance for SDK extensions or extensions using the SDK module loader, added download protection for a large number of executable file types, added option to Find in page that allows users to limit search to whole words only, and more. See the release notes for details.

Comments (27 posted)

Announcing Rust 1.13

Rust 1.13.0 has been released. "The 1.13 release includes several extensions to the language, including the long-awaited ? operator, improvements to compile times, minor feature additions to cargo and the standard library. This release also includes many small enhancements to documentation and error reporting, by many contributors, that are not individually mentioned in the release notes. This release contains important security updates to Cargo, which depends on curl and OpenSSL, which both published security updates recently."

Comments (none posted)

xorg-server 1.19.0

X.Org Server 1.19 has been released. This version features threaded input support, XWayland improvements, modesetting driver improvements, and more.

Full Story (comments: none)

Development newsletters

• Emacs News (November 14)
• These Weeks in Firefox (November 16)
• What's cooking in git.git (November 9)
• What's cooking in git.git (November 11)
• What's cooking in git.git (November 16)
• Git Rev News (November 16)
• GNU Toolchain Update (November)
• This week in GTK+ (November 14)
• OCaml Weekly News (November 15)
• Perl Weekly (November 14)
• Python Weekly (November 10)
• Ruby Weekly (November 10)
• This Week in Rust (November 15)
• Wikimedia Tech News (November 14)

Comments (none posted)